Q & A
Data Management
Q: Can you describe the process your team follows to securely store, process, and analyse large volumes of data?
Our group has a systematic way to safely store, process and analyse large amounts of data, and at the same time make sure the data will remain secure while being processed efficiently. This is a high-level summary of the methods that are used:
1. Data Gathering/Collection
We gather information from many different platforms and devices that connect to a larger network. These include things such as the Internet of Things (IoT) where sensor devices recording all types of environmental information to databases that hold vast amounts of information. We securely (via coded transfer) compile collected information together, in one central location.
2. Secure Data Storage
We store the collected information in a very safe and secure location or environment using locked storage solutions (in the form of either cloud storage or physical on-premises storage), along with access and security protocols and frequent audits of the storage solution to prevent unauthorized users from accessing stored information. Data is segregated according to the data’s level of sensitivity.
3. Data Formatting/Preparation
We prepare the collected raw data for use by first cleaning and transforming the raw data, by identifying all missing data elements, removing duplicate records, and standardizing on the same format. After this process has been completed, the data is considered clean and ready for further analysis purposes.
4. Data Processing through Scaled Out Frameworks
We utilize distributed computing technologies like Hadoop and Apache Spark, to process large amounts of data efficiently. Both frameworks allow us to have additional processing power in accordance with the volume of data needing processed. Both technologies provide for efficient data processing of very large datasets.
5. Data Analysis and Data Visualization
In the end, we use advanced analytics methodologies, such as predictive or statistical analytics, as well as artificial intelligence/machine learning models on the processed data to derive insights and actionable intelligence and build out visualizations or dashboards to present complex processed data in a way that makes sense to those using it.
