Really awesome seeing more people work on this! I’m one of the founders of Opik https://github.com/comet-ml/opik which does similar things but also has a UI and supports massive scale. Curious to hear if you have any feedback!
Excited to see more people building in this space. From what we've seen with customers it's critical to be able to compare what you're seeing in production to what you trained on (rather than historical period). That's almost the textbook definition of drift. Do you have a sense on how to approach that?
At Comet.com (disclaimer: i'm the CEO/Co-founder) we provide experiment tracking and artifacts management so we have the training distributions for comparison. I'm always curious how it looks like for a monitoring only solution
Completely agree! We have also seen our users more concerned about comparing the real-world distribution against the training data as compared to previous month's data (we found latter is more useful for PMs and setting alerts).
We currently allow users to specify their training data in the config which is used to initialise the UpTrain framework (in form of json file but are planning to support pytorch/tf data-loaders). In the background, the tool does all the binning and clustering to convert these continuous variables into discrete buckets to later calculate divergence, which is then used to quantify drift.
Thanks for the very relevant comment :) We provide users the option to attach their training data from csv/json (working to support loading from cloud storage provider or data lakes). We have illustrated this in some of our examples, such as the human orientation classification: https://github.com/uptrain-ai/uptrain/blob/main/examples/hum...
Hi HN! Excited to share with you our recent open source project CodeCarbon - a collaboration between Mila, BCG GAMMA, Haverford College, and Comet.ml. CodeCarbon records the amount of power being used by the underlying infrastructure from major cloud providers and privately hosted on-premises data centers. Based on publicly available data sources, it estimates the amount of CO₂ emissions produced by referring to the carbon intensity from the energy mix of the electric grid to which the hardware is connected.
Hey! I'm one of the founders at Comet.ml. We believe that Git should continue to be the approach for managing code (similar to dvc) but we adapted it to the ML workflow. Our approach is to compute a git patch on every run so later you can 'git apply' if you'd like (https://www.comet.ml/docs/user-interface/#the-reproduce-butt...).
Streamlit is awesome but solves a different use case. There's definitely similarities but also a few core differences. 1. Comet Panels reads the experimentation data from our API/Database. Streamlit is much more like Jupyter notebooks where you'd have to setup your own data store and implement logic to update it with new experiment data.
This also means that if you pull a visualization from the gallery it should work out of the box as the API is standard among everyone!
2. Comet Panels is freely hosted so no need to setup a webserver
3. Comet Panels is natively written in JS/HTML/CSS so you can use any JS library out of the box. Streamlit is written in Python and you might be limited to the built in UI components (or you can build a custom on which is also in JS/HTML/CSS)
Hi HN! Really excited to share this with with those of you training ML models. Comet Panels is an easy and free way to build visualizations, widgets and apps in HTML/JS/CSS on top of your ML experiment data and a gallery where you can share your work with the world.
Panels is built on our experiment management platform (100% free for individuals). Essentially as a user you can report all of your metrics, params, assets, data samples etc. Everything stored is also accessible via our JS SDK so it's really easy to fetch data for your panels.
The main idea is that visualizations aren't stale. Every time a new experiments is reported your visualization will update with that additional data.
Sure, that demonstrates the issue.
Problem is with using train_test_split(X, yy, test_size=0.2..) - this assumes independent samples, which is violated for this dataset (because some come from same source audio files). The easiest (and completely acceptable) is to use one fold as the validation data, one fold for the test set, and the remaining folds as training.
This problem is unfortunately quite common even in academic papers using this dataset, even though the authors warn about it.
EDIT: There is one more issue with Urbansound8k folds, and that is that the difficulty of the various folds is quite different. So one should ideally report the performance across all folds (mean/std or boxplot). But this is a minor issue compared to data leakage.
PS: Nice use of Comet.ml platform this, collaborating online on improving the experimental setup :)
Hey jononor — we've updated the post to split the training and test sets based on the folds. Good catch and thanks again for reporting this. Some of the experiments in the project will still have the old code, but the blog post will reflect this new train/test split.
Hi M0zg! Gideon from Comet- sorry to hear you're having issues. Did you every try to report these? if you share more info at support@comet.ml or at our slack channel i'm sure we can fix it / improve. On a general note: 1. you can see min/max values in the metrics tab for finished/running experiments. 2. we spend tons of time on performance but these are actually difficult problems to solve, i.e if you have ten charts all showing 10k data points all updating in real time. That said if you share your project we can use it to improve. Finally the SDK is designed to never crash or slow down your training and this is the first time we've heard that complaint - again please ping us so we can figure out what's going on.
Really excited to see this relaunched. Every DS team has issues around dataset management. We previously shared a tutorial on how to get a fully reproducible pipeline with Quilt + Comet.ml https://blog.quiltdata.com/building-a-fully-reproducible-mac...
