Matias Carrasco Kind @ NCSA

DES Collaboration Phone Call, September 12th, 2016

Data Access for Astronomical Surveys

A brief description and demo of tools @ DESDM

DES Collaboration Phone Call


Matias Carrasco Kind @ NCSA/UIUC

Introduction


  • DES : Gold (Data) Mine

  • Excellent Job at mining the data

  • Consumer outside the mine

  • Need to bring gold (data) outside

  • Tools and interfaces

  • Getting ready for Y3!

Outline

DES Science Portal

desportal.cosmology.illinois.edu


  • @ NCSA by managed by LIneA

  • Interactive Coadd Image exploration

  • Moving some services from old-portal to the Science portal

  • Already uploaded with Y3A1 Test data

easyaccess        

easyaccess.png

Motivations

  • DES DB in Oracle

  • Needed upgrade from older versions (trivialaccess)

  • Enhanced SQL command line interpreter

  • Astronomically friendly

  • Python API, web interface

  • There are many other clients but needed one standard for beginners

Some cool features

  • History of commands
  • Smart TAB auto-completion of commands, columns, users, tables and path
  • Very flexible configuration
  • Completely open source and written in python
  • Write tables in csv, tab, fits and hdf5 formats
  • Load sql queries directly from a file
  • Load table directly in to DB in one step, no memory issues
  • Can be imported as python module (py2/py3)
  • Tables are handled and displayed using pandas
  • Can run python functions inside query (beta)
  • Many more...

Installation

Making things even easier

  • Clone it from github
  • Using pip or easy_install
  • Using EUPS @ DESDM
  • Pre-installed in Docker Image (and run outside the container)
  • New! Install with conda

conda install easyaccess -c mgckind

Future and Demo

  • Generalization (SQLAlchemy), in-query python functions, batch jobs, checkpoint, web, better Jupyter integration, more...

  • easyacess: Enhanced SQL interpreter for astronomical surveys (Carrasco Kind+, 2016?)

  • For a more detailed tutorial check Here

easyaccess        

Some DESDM statistics

(Scroll down )

DESDM New accounts


More than 220 New users since new On-boarding!


New DES Member? Check Here


Several Services: DB, Confluence, HipChat, BitBucket, Stackoverflow


DB Users


Doubled number of users since easyaccess


More than 180 users with easyaccess with different activity levels


Feedback via github issues, stack-overflow-like questions, email list and personal inquiries

DB Quotas


Personal space actively used and constantly increasing


People create and share tables with one command


~ 4.5 TB in personal space, tables and index tables included

SV Release

https://des.ncsa.illinois.edu/internal/summary

Experimental UI projects being developed (Openstack + Docker)

deslabs.ncsa.illinois.edu

Experimental UI projects being developed (Openstack + Docker)

deslabs.ncsa.illinois.edu

JupyterHub + easyaccess + DS9



  • Multi-user jupyter notebook sessions
  • Access with DB credentials
  • Running in OpenStack ("The Cloud")
  • Easy setup (Running from Docker container)
  • Easyaccess and other tools installed
  • http://141.142.236.167:8000/hub/
  • Local JS9 (DS9 in Javascript) integration
  • Still in beta mode, planning to scale up
  • Open to DES collaborators
Live demo!

Online Cutout Services

descut.cosmology.illinois.edu

  • Cutout Services for both: Coadd images and Single Epoch

  • Nice REST API so can be also accessed from e.g. python

  • Almost production-like services with GPFS mounted

  • Soon to be "oficcially" released, we are merging both services



Live demo!

Coadd image cutouts example

Integrated tools demo

Conclusions

  • Tools being developed to make data access easier

  • Easyaccess, Cutouts, Jupyter, Js9, Science Portal, etc...

  • Opensource and integrated software

  • Bring software to the data!

  • Want to help? Want some infrastructure? Let me know!

deslabs.ncsa.illinois.edu

Thanks!


Questions?

Matias Carrasco Kind

mcarras2@illinois.edu

matias-ck.com
deslabs.ncsa.illinois.edu