DataPusher

Overview

DataPusher is an extension for CKAN (Comprehensive Knowledge Archive Network) that automates the process of converting uploaded tabular data (e.g., CSV files) into a more usable format by storing it in a CKAN datastore. This enables advanced querying, filtering, and visualization directly from the CKAN interface. DataPusher enhances CKAN’s capabilities by ensuring that datasets are immediately ready for exploration and analysis after they are uploaded.

Key Features

  • Automatic Data Processing:

Automatically processes and converts tabular data formats like CSV into the CKAN datastore. Handles data processing in the background, allowing users to continue working without interruption.

  • Integration with CKAN Datastore:

Stores processed data in the CKAN datastore, making it accessible through CKAN’s API. Supports querying, filtering, and visualization of data within the CKAN interface.

  • Error Handling and Logging:

Provides detailed logging of the data processing tasks, making it easier to troubleshoot issues. Automatically retries processing if errors occur, ensuring data is successfully loaded whenever possible.

  • Scalability:

Designed to handle large datasets efficiently, processing them in chunks if necessary. Scalable to meet the needs of organizations with significant data volumes.

Use Cases

  • Simplified Data Uploads:

Users upload CSV files, and DataPusher automatically processes and stores the data, reducing the manual effort required. Ensures that data is immediately available in a structured format, ready for use in applications and visualizations.

  • Enhanced Data Querying:

DataPusher enables advanced querying capabilities by converting raw CSV data into a format that supports filtering and analysis. Users can interact with data directly within CKAN, without needing to download and manipulate files locally.

  • Automated Data Processing:

Organizations use DataPusher to automate the processing of incoming data, ensuring it is consistently prepared and available in the datastore. Ideal for environments with frequent data uploads, such as government open data portals or research data repositories.

  • Improved Data Quality:

By automating the data processing, DataPusher reduces the likelihood of human errors during data preparation. Ensures consistent formatting and structure of datasets, which is critical for accurate analysis and reporting.

DataPusher docs