intelligent decision superiority through vision and language technology
Key Points
- DAIRNet Theme: Autonomous Processing and Reasoning
- University of Adelaide (Australian Institute of Machine Learning)
- Point of contact: Dr Jamie Sherrah
Defence has a data deluge problem. There are increasing numbers of sensors and information sources, increasing detail within information, unstructured and uncertain data, a mix of data types and timeframes, and a fixed, relatively small number of ADF personnel. How can the analyst use all of the information for dynamic tasks within the required timeframe?
A natural language interface to geospatial and imagery databases can solve this problem for geospatial analysts and decision-makers. Vision and language machine learning can be used to answer questions of geo-spatial data across many images and over large areas. It can process, filter and combine information into summarised forms, analyse and learn from data at scale and speed, interpret and fuse heterogeneous and unstructured data. It can also preform new tasks that weren’t designed into the system and present uncertainty and explanations to build trust.
Visual Question Answering (VQA), and subsequently Visual Dialogue (VD), represent the closest analogues of the Intelligent Decision Support problem in the Machine Learning literature. These challenges require that an agent amalgamate information from multiple modalities in real time to collaborate with a human operator towards achieving a complex and weakly specified goal.
We propose an approach to achieving intelligent decision support that would see human and machine collaborate through a natural language interface. The system would analyse large volumes of ISR data of various modalities, and support the analyst/pilot/warfighter in achieving their goal.
If successful, this research could result in systems that rapidly analyse large volumes of ISR data of various modalities and support the analyst/pilot/warfighter in achieving their goal, without requiring either the goal or the interactions to be specified beforehand.
publications
- Felix, R, Repasky, B, Hodge, S, Zolfaghari, R, Abbasnejad, E & Sherrah, J 2021, ‘Cross-Modal Visual Question Answering for Remote Sensing Data’, The International Conference on Digital Image Computing: Techniques and Applications (DICTA 2021), pp. 1-9. DOI: 10.1109/DICTA52665.2021.9647287
