View our Web Map here and its code here
View the R Markdown as it currently stands at this link
View the tracking sheet here
1. The Idea
This project by the Datameet Pune group has set out a goal to make village level spatial data analysis more accessible.
Imagine.. not being limited to coarse taluka or district level data but being able to explore spatial patterns and temporal
trends with the census village as the unit of analysis. What if we could understand how a village compares to its neighbouring villages
in population, social composition, economic status, nature of agriculture, soil, physical infrastructure and many other factors. What
if maps allowed us to understand precisely which villages fall under which assembly constituencies? What could this mean for political
accountability? What if we could capture real time village level trends with crowdsourced data for more effective reporting? Could this
complement and help overcome limitations of mainstream media coverage of rural development..
2. The Background
To put together good village level maps we needed just two things: village boundaries in a geospatial data format and village level data.
We set out with the goal of taking village level boundaries made available by
Bhuvan Panchayat the Government of India's spatial data portal for rural development. For village level data, the most easily accessible dataset
was the Census Districts Handbooks. For more details on both these, see below.
Datasets we have:
1. 2011 District Census handbook (DCHB) Maharashtra [xlsx and csv format]
The census has published district census handbooks available openly online which contains information on hundreds of variables (demographic, social, economic, access to water, sanitation, health, education, ICT, transport etc) for all of Maharashtra's approximately 44000 villages.
You can go to this link and click on Maharashtra, village amenities to get the large excelsheet for all of Maharashtra (72mb) or download a csv for Maharashtra (38mb) or single district wise csv (<3mb) from this google drive link
2. Census Code Lookup Tables [csv format]
The Census codes for villages in India have not remained the same in going from census 2001 to 2011. The coding scheme was changed and lookup tables of 2001 codes and their corresponding 2011 codes are accessible on the Govt of India egov platform via this link. Note that this egov site is often problematic, alternatively you can go to this google drive link for the same file
3. Village Boundaries for Maharashtra [shapefile format and their attribute tables in csv format]
We have two versions of this shapefile and two versions of their attribute tables. Both were obtained from Bhuvan Panchayat, by accessing their WFS endpoint via using the gdal library.
Version 1: The first one has approximately 44,000 villages, it has only 2001 codes: district, sub-district, village. It does not have village names, or any 2011 codes. Hence the problem. We cannot easily append 2011 census data to this file to prepare a map.
Version 2: The second file has both 2001 and 2011 codes, district, sub-district and village and also village names. However it shows approximately 48,000 villages, many more than those existing in the 2011 DCHB.
3. The Process
The Datameet Pune group consists of about 15-20 volunteers with widely varying skillsets, in data analysis and visualization and also volunteers associated with different non-profit organisations who do grounded work in rural development. Through regular meetups they have collaborated and put together their skills to develop a better understanding of rural issues through this data.
The main tasks undertaken were..Generating meta-data
Effectively and reliably using the village maps requires performing a quality check on them. Since very little meta-data was available with these boundary files this was important. We sought to answer questions such as are all villages accounted for, are any missing, any duplicated? Are the villages located in the correct talukas and districts? etc. To help answer these questions we worked with the open source softwares R and QGIS, and developed a few R scripts for this. View the R Markdown as it currently stands at this link
Developing web maps
Besides this we also developed a web map, using Leaflet to give users a feel of what the maps make possible. The web maps use Census 2011 village level data. View the web map here and its code here
4. Plans going ahead
Major plans going ahead are to improve on the R scripts and web maps and also outreach to publicize the maps widely. More specifically..
- Include 2001 census variables in the web maps as well
- Get users familiar with the tracking sheet so they can report errors
- Think about ways to display multiple variables e.g. with simple logical/math operations
- Focus on making these maps available in Marathi. This is still a work in progress, we welcome anyone who wishes to contribute.