Apex Code Development You need to sign in to do that
Don't have an account?
Jason Durheim Can you separate Einstein training data from the test data?
This may simply just not be available yet, didn't see it in any of the doc surrounding training a model.
You can specify what percentage of the dataset gets used for training with the remainder being used for testing. Is there any mechanism to explicitly define a dataset to use for training and another dataset to be used for testing?
It would seem like this would be valuable for tuning the model, having a static test data set to always test against while being able to modify the training dataset to increase the accuracy.
You can specify what percentage of the dataset gets used for training with the remainder being used for testing. Is there any mechanism to explicitly define a dataset to use for training and another dataset to be used for testing?
It would seem like this would be valuable for tuning the model, having a static test data set to always test against while being able to modify the training dataset to increase the accuracy.
I understand that you can vary the percentage used for the test set, what I was getting at was more about the fact that I don't know (using the default as an example), which 10% of the test data is used as the test set versus the other 90% as the training set. As it goes through the training iterations is this 10% consistent? Or does it do some number of iterations on a particular 10% until optimal accuracy is achieved and then go through a new 10%, etc, to try and further refine the accuracy across that data set?
To put it another way, with the current APIs it's hard to try and tune the accuracy for certain categories of questions if you aren't sure you have a consistently repeatable test set to compare changes in the training set to. In our light PoC, even if Einstein was getting the correct answer, it might have only been 60% sure, and we were looking to see if we could improve on that number. In other cases we saw that the ordering of the words in the question or the presence of specific words in the sentence might greatly change the probability, even to the point of changing the top returned answer. We also tried to play around with longer sentences versus shorter ones (even single words in some cases) to try and adjust the accuracy, but was a little harder sometimes to see how those changes to the dataset affected the accuract of certain groups of questions.
A large part of my question probably comes from the fact that I've worked internally on other AI platforms in the past, so I'm used to a setup with an explicitly separate training set and test set and knowing the accuracy results. Maybe that partially qualifies me as a "data-scientist" where I want to see and work more with the guts of the training process, when Einstein is currently more designed for ease of accessability without having to go to that level. So totally nothing wrong with these capabilities not being available (maybe even ever), was mostly just curious if there was any thought that there may be some 'advanced options' that might allow for a litle more low-level work with tuning the accuracy of Einstein :)
Thanks!
As to your examples... Those are great ways to test and enhance your model. Since we use a Neural Net architecture, we aren't simply counting words; with that word order, synonymns, size/length of the utterance, etc... all add to variables that can be leveraged to enlarge your training data and enhance your models.
Glad to hear that we are checking off a lot of your requirements but do let us know how we can ensure success for your model tests. We are constantly thinking of ways to provide more granular model metrics and visibility into potential errors in predictions.
Thanks!