An open source library for deep learning end-to-end dialog systems and chatbots.
DeepPavlov is an open-source conversational AI library built on TensorFlow and Keras. It is designed for
As of version 0.1.0
all models, embeddings and other downloaded data for provided configurations are
by default downloaded to the .deeppavlov
directory in current user’s home directory.
This can be changed on per-model basis by modifying
a ROOT_PATH
variable
or related fields one by one in model’s configuration file.
In configuration files, for all components, dataset readers and iterators "name"
and "class"
fields are combined
into the "class_name"
field.
deeppavlov.core.commands.infer.build_model_from_config()
was renamed to build_model
and can be imported from the
deeppavlov
module directly.
The way arguments are passed to metrics functions during training and evaluation was changed and documented.
Import key components to build HelloBot.
from deeppavlov.skills.pattern_matching_skill import PatternMatchingSkill
from deeppavlov.agents.default_agent.default_agent import DefaultAgent
from deeppavlov.agents.processors.highest_confidence_selector import HighestConfidenceSelector
Create skills as pre-defined responses for a user’s input containing specific keywords or matching regexps. Every skill returns response and confidence.
hello = PatternMatchingSkill(responses=['Hello world!'], patterns=["hi", "hello", "good day"])
bye = PatternMatchingSkill(['Goodbye world!', 'See you around'], patterns=["bye", "chao", "see you"])
fallback = PatternMatchingSkill(["I don't understand, sorry", 'I can say "Hello world!"'])
Agent executes skills and then takes response from the skill with the highest confidence.
HelloBot = DefaultAgent([hello, bye, fallback], skills_selector=HighestConfidenceSelector())
Give the floor to the HelloBot!
print(HelloBot(['Hello!', 'Boo...', 'Bye.']))
Jupyther notebook with HelloBot example.
Components
Named Entity Recognition | Slot filling |
Intent/Sentence Classification | Question Answering over Text (SQuAD) |
Sentence Similarity/Ranking | TF-IDF Ranking |
Morphological tagging | Automatic Spelling Correction |
Skills
Goal(Task)-oriented Bot | Seq2seq Goal-Oriented bot |
Open Domain Questions Answering | eCommerce Bot |
Frequently Asked Questions Answering | Pattern Matching |
Embeddings
ELMo embeddings for the Russian language
FastText embeddings for the Russian language
Auto ML
Tuning Models with Evolutionary Algorithm
Linux
and Windows
platforms and Python 3.6
Python 3.5
is not supported!Windows
platform requires Git
for Windows (for example, git), Visual Studio 2015/2017
with C++
build tools installed!Python 3.6
:
virtualenv env
Linux
source ./env/bin/activate
Windows
.\env\Scripts\activate.bat
pip install deeppavlov
Demo of selected features is available at demo.ipavlov.ai
To use our pre-trained models, you should first install their requirements:
python -m deeppavlov install <path_to_config>
Then download the models and data for them:
python -m deeppavlov download <path_to_config>
or you can use additional key -d
to automatically download all required models and data with any command like interact
, riseapi
, etc.
Then you can interact with the models or train them with the following command:
python -m deeppavlov <mode> <path_to_config> [-d]
<mode>
can be train
, predict
, interact
, interactbot
, interactmsbot
or riseapi
<path_to_config>
should be a path to an NLP pipeline json config (e.g. deeppavlov/configs/ner/slotfill_dstc2.json
)
or a name without the .json
extension of one of the config files provided in this repository (e.g. slotfill_dstc2
)For the interactbot
mode you should specify Telegram bot token in -t
parameter or in TELEGRAM_TOKEN
environment variable. Also if you want to get custom /start
and /help
Telegram messages for the running model you should:
metadata.labels.telegram_utils
parameter with name which refers to the added section of utils/settings/models_info.jsonFor the interactmsbot
mode you should specify Microsoft app id in -i
and Microsoft app secret in -s
. Also before launch you should specify api deployment settings (host, port) in utils/settings/server_config.json configuration file. Note, that Microsoft Bot Framework requires https
endpoint with valid certificate from CA.
Here is detailed info on the Microsoft Bot Framework integration
You can also store your tokens, app ids, secrets in appropriate sections of utils/settings/server_config.json. Please note, that all command line parameters override corresponding config ones.
For riseapi
mode you should specify api settings (host, port, etc.) in utils/settings/server_config.json configuration file. If provided, values from model_defaults section override values for the same parameters from common_defaults section. Model names in model_defaults section should be similar to the class names of the models main component.
Here is detailed info on the DeepPavlov REST API
All DeepPavlov settings files are stored in utils/settings
by default. You can get full path to it with python -m deeppavlov.settings settings
. Also you can move it with with python -m deeppavlov.settings settings -p <new/configs/dir/path>
(all your configuration settings will be preserved) or move it to default location with python -m deeppavlov.settings settings -d
(all your configuration settings will be RESET to default ones).
For predict
you can specify path to input file with -f
or --input-file
parameter, otherwise, data will be taken
from stdin.
Every line of input text will be used as a pipeline input parameter, so one example will consist of as many lines,
as many input parameters your pipeline expects.
You can also specify batch size with -b
or --batch-size
parameter.
We have built several DeepPavlov based Docker images, which include:
riseapi
mode).Here is our DockerHub repository with images and deployment instructions.
Jupyter notebooks and videos explaining how to use DeepPalov for different tasks can be found in /examples/tutorials/
DeepPavlov is Apache 2.0 - licensed.
If you have any questions, bug reports or feature requests, please feel free to post on our Github Issues page. Please tag your issue with bug
, feature request
, or question
. Also we’ll be glad to see your pull requests to add new datasets, models, embeddings, etc.
DeepPavlov is built and maintained by Neural Networks and Deep Learning Lab at MIPT within iPavlov project (part of National Technology Initiative) and in partnership with Sberbank.