DataPulse MY

Live dashboard: https://r3dz4r.github.io/datapulse-my/

🤖 AI-agent-ready — DataPulse MY is built for AI. An agent fetches one file (llms.txt) and can use all 122 official Malaysian datasets immediately — including + 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY) — no integration, no scraping, licences declared, with an honest eight-status trust taxonomy instead of a blanket green checkmark.

DataPulse MY is an open-source trust layer for Malaysian public data. It makes official datasets easier to assess and reuse by publishing a small manifest, human-readable health reports, and machine-readable health envelopes.

It does not replace the official source. It documents what is available, whether it is fresh, how its schema behaves, and which collection quirks users need to handle.

Who it is for

• Journalists and researchers checking whether a public dataset is usable.
• Civic technologists building reproducible data pipelines.
• Public servants improving the discoverability and reliability of open data.
• Developers who need stable, machine-readable dataset health metadata.

Use this for

• Journalist fact-check: before citing a fuel price figure, check fuelprice freshness to make sure it's current.
• Pipeline health gate: fail the build if data.gov.my has been down for more than 24 hours.
• RAG knowledge base: consume the JSON envelopes as structured context for a chatbot answering "what's the latest BNM rate?"

Dataset health

Health is reported as fresh, aging, stale, degraded, browser-dependent, unreachable, unknown, or unknown-freshness. The last value means the URL and content shape work, but neither a Last-Modified header nor a parseable content date proves when the data was updated. The public _trust_summary shows the distribution and explicitly counts missing freshness and row-count signals.

Current distribution (_trust_summary): 80 fresh · 25 aging · 11 stale · 5 browser-dependent · 1 unreachable

Subscribe: RSS feed — get notified when dataset health changes.

AI-agent-ready — what it means for you

Give your organisation's AI tools current, licensed, and verified Malaysian public data without first building a custom integration. DataPulse MY makes the full portfolio discoverable from one self-describing index, ready for agents, RAG systems, and internal knowledge tools to consume.

What being AI-ready gives you

• Zero integration work: an AI agent or LLM/RAG system fetches one llms.txt and can use the entire portfolio immediately — no scraping, API-key setup, or data-format reverse-engineering.
• Honest freshness signals: automatic weekly monitoring separates HTTP reachability, browser dependency, schema validity, and source freshness so missing evidence is visible instead of being labelled healthy.
• Machine-readable and licence-clear: every dataset has a JSON envelope with its schema, licence, and refresh cadence, giving legal and engineering teams the information they need to approve and integrate it.
• Trustworthy for AI: verified official sources and explicit licences let agents cite and use the data without permission ambiguity.
• RAG and knowledge-base ready: drop the envelopes into a retrieval pipeline to ground chatbots and AI tools in current Malaysian public data.

122 datasets, 110 under CC BY 4.0, 12 under OGL, each assessed with the honest eight-status trust taxonomy.

MCP server (read-only)

DataPulse MY also exposes an AI-ready, read-only MCP server so agents can query the catalogue natively:

• Endpoint: https://mcp.data-pulse.my/mcp (Streamable HTTP, no auth)
• 5 tools: search_datasets, get_dataset, find_stale, get_provenance, find_by_licence
• 3 resources: datapulse://index, datapulse://licences, datapulse://{dataset_id}

The public endpoint is live and serves all 5 read-only tools over the 122-dataset catalogue.

Connect from Claude Desktop:

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

See llms.txt for the full discovery index, and docs/mcp-deploy.md for the deployment architecture.

How to consume the data

Verify access:

curl -s https://r3dz4r.github.io/datapulse-my/llms.txt

• llms.txt — curated dataset index
• datapulse.json — manifest with a declared $schema
• health/latest.json — latest freshness snapshot
• feed.xml — dataset health change feed
• datapulse.schema.json — manifest schema

To consume the portfolio:

1. Fetch llms.txt for the curated index.
2. Fetch datapulse.json for the machine-readable manifest, including licence, refresh cadence, and geographic coverage.
3. Fetch health/latest.json to check freshness before use.
4. Cite each dataset according to its licence and attribution requirements.

robots.txt allows all agents; scripts/verify_agent_ready.sh is the agent-consumer self-test.

Included datasets

• Malaysian Fuel Prices — Samples: CSV, JSON
• ePerolehan Tender Notices (DIIKLANKAN) — Sample JSON
• PriceCatcher (Daily Grocery Prices) — Samples: main CSV, item lookup, premise lookup

Daily Reference Data (Bank Negara Malaysia)

These four BNM reference-rate datasets are updated on weekdays at fixed MYT publication times:

• BNM Daily Exchange Rates (0900) — Sample JSON

• BNM Daily Exchange Rates (1130) — Sample JSON

• BNM Daily Exchange Rates (1200) — Sample JSON

• BNM Daily Exchange Rates (1700) — Sample JSON

• MET Malaysia Weather Forecast — Samples: CSV, JSON

• DOE APIMS Air Quality (Hourly) — Sample JSON

• DOE RQIMS River Water Quality (Continuous) — Sample JSON

• DOE MQIMS Marine Water Quality (Monthly) — Sample JSON

• KKM iDengue Weekly Dengue Cases — Sample JSON

• OpenDOSM Crime by District (Annual) — Sample JSON

• OpenDOSM Monthly CPI by State — Sample JSON

• OpenDOSM Annual Real GDP by State — Sample JSON

• OpenDOSM Quarterly Real GDP — Sample JSON

• OpenDOSM Annual Real GDP by Supply Sector — Sample JSON

• OpenDOSM Monthly Trade Headline — Sample JSON

• OpenDOSM Monthly CPI Inflation by Division — Sample JSON

• OpenDOSM Monthly Trade by End Use (BEC) — Sample JSON

• OpenDOSM Quarterly Labour Force Statistics — Sample JSON

• OpenDOSM Quarterly Labour Force Statistics by State — Sample JSON

• OpenDOSM Annual Employment by Sector and Sex — Sample JSON

• OpenDOSM Annual Population by State — Sample JSON

• OpenDOSM Annual Nominal GDP by Supply Sector — Sample JSON

• OpenDOSM Quarterly Nominal GDP — Sample JSON

• OpenDOSM Quarterly Real GDP (Seasonally Adjusted) — Sample JSON

• OpenDOSM Annual Nominal GDP and GNI — Sample JSON

• OpenDOSM Monthly Core CPI Inflation by Division — Sample JSON

• OpenDOSM Monthly CPI Inflation by State and Division — Sample JSON

• OpenDOSM Monthly Producer Price Index — Sample JSON

• OpenDOSM Annual Labour Force Statistics — Sample JSON

• OpenDOSM Monthly Labour Force Statistics — Sample JSON

• OpenDOSM Monthly Trade by SITC Section — Sample JSON

• OpenDOSM IPI for Export-Oriented Divisions — Sample JSON

• OpenDOSM IPI for Domestic-Oriented Divisions — Sample JSON

• data.gov.my Monthly Interest Rates — Sample JSON

• data.gov.my Quarterly Federal Government Revenue — Sample JSON

• data.gov.my Quarterly Federal Operating Expenditure — Sample JSON

• data.gov.my State Government Expenditure — Sample JSON

• data.gov.my Monthly Money Aggregates — Sample JSON

• data.gov.my Monthly Currency in Circulation — Sample JSON

• data.gov.my Monthly Payment Systems — Sample JSON

• data.gov.my Monthly Payment Instruments — Sample JSON

• data.gov.my Monthly Payment Channels — Sample JSON

• data.gov.my Annual Interest Rates — Sample JSON

• data.gov.my Annual EPF Dividend Rates — Sample JSON

• data.gov.my Monthly Vehicle Registrations by Type and Fuel — Sample JSON

• data.gov.my Daily FPX Transactions — Sample JSON

• OpenDOSM Annual Births by State — Sample JSON

• OpenDOSM Annual Deaths by State — Sample JSON

• OpenDOSM Annual Maternal Deaths by State — Sample JSON

• OpenDOSM Annual Marriages by State and Sex — Sample JSON

• data.gov.my Hospital Beds by State and Hospital Type — Sample JSON

• data.gov.my Healthcare Staff by State and Staff Type — Sample JSON

• data.gov.my Daily Blood Donations by Blood Group and State — Sample JSON

• data.gov.my Infant Immunisation Coverage — Sample JSON

• data.gov.my Sexually Transmitted Diseases by State — Sample JSON

• data.gov.my Daily PeKaB40 Health Screenings by State — Sample JSON

• data.gov.my Malaysian National Health Accounts Expenditure — Sample JSON

• data.gov.my Monthly Electricity Consumption — Sample JSON

• data.gov.my Monthly Electricity Supply — Sample JSON

• data.gov.my Water Consumption by State and Sector — Sample JSON

• data.gov.my Water Production by State — Sample JSON

• data.gov.my Access to Treated Water by State and Strata — Sample JSON

• data.gov.my Daily Public Transport Ridership — Sample JSON

• data.gov.my Monthly KTMB Ridership — Sample JSON

• data.gov.my Cellular Subscribers by Plan Type — Sample JSON

• data.gov.my Prisoners by State and Sex — Sample JSON

• data.gov.my Drug Addicts by State and Age Group — Sample JSON

• data.gov.my Female Representation in Local Authorities — Sample JSON

• data.gov.my Female Representation in Parliament — Sample JSON

• data.gov.my Monthly Marine Fish Landings by State — Sample JSON

• data.gov.my Crop Area and Production by State — Sample JSON

• data.gov.my Public Education Institutions by District — Sample JSON

• OpenDOSM Household Income, Malaysia — Sample JSON

• OpenDOSM Household Income by State — Sample JSON

• OpenDOSM Household Income by District — Sample JSON

• OpenDOSM Poverty, Malaysia — Sample JSON

• OpenDOSM Poverty by State — Sample JSON

• OpenDOSM Poverty by District — Sample JSON

• OpenDOSM Income Inequality, Malaysia — Sample JSON

• OpenDOSM Income Inequality by State — Sample JSON

• OpenDOSM Income Inequality by District — Sample JSON

• OpenDOSM Household Expenditure by DUN — Sample JSON

• OpenDOSM Household Expenditure by Parliamentary Constituency — Sample JSON

• OpenDOSM Annual Population, Malaysia — Sample JSON

• OpenDOSM Annual Population by Parliamentary Constituency — Sample JSON

• OpenDOSM Annual Deaths by District and Sex — Sample JSON

• OpenDOSM Annual Marriages by State, Age, and Sex — Sample JSON

• OpenDOSM Annual Fertility — Sample JSON

• OpenDOSM Annual Maternal Deaths, Malaysia — Sample JSON

GTFS transit feeds

The transport namespace adds 16 GTFS Static schedule ZIPs and 14 GTFS Realtime vehicle-position protobuf feeds for KTMB, Prasarana, and BAS.MY services. Static samples are under samples/gtfs-static/ and realtime snapshots are under samples/gtfs-realtime/.

DataPulse MY currently tracks 122 datasets in total.

Current coverage

Refresh cadence

How to use it

Start with datapulse.json to discover datasets and their official sources. Follow each health_report link for a plain-language assessment, or consume the matching file under data/json/ in an automated workflow.

For example, a data pipeline can inspect status, content_freshness_date, and freshness_signal_source before processing a source, while a researcher can review the known quirks before designing a collection method.

Roadmap

• Scheduled health checks — weekly.
• RSS feed — available.
• Status badges — available.
• More datasets — planned.

Adopt a dataset

Know a Malaysian public dataset that deserves dependable health metadata? Adopt it: verify its source and licence, document its schema and quirks, and submit a health report. See CONTRIBUTING.md for the expected three-file contribution model.

New contributors can start with the repository's Good first issues or propose a dataset through the GitHub issue forms. Maintainers use good first issue (yellow), adopt-a-dataset (blue), freshness-check (blue), bug (red), documentation (blue), question (purple), and wontfix (gray) to route contributions.

Licence

DataPulse MY is released under the MIT License. Source datasets remain subject to the licences and attribution requirements stated in their individual health reports.