datadiligence package

Subpackages

Submodules

datadiligence.bootstrap module

functions to preload default evaluators and make accessible globally

datadiligence.bootstrap.deregister_evaluator(name)[source]

Deregister an evaluator.

Parameters:

name (str) – The name of the evaluator.

datadiligence.bootstrap.filter_allowed(name=None, **kwargs)[source]

Filter a list of content.

Parameters:

  • name (str) – The name of a specific evaluator.

  • **kwargs – Arbitrary keyword arguments to read args from.

datadiligence.bootstrap.get_evaluator(name)[source]

Get an evaluator.

Parameters:

name (str) – The name of the evaluator.

datadiligence.bootstrap.is_allowed(name=None, **kwargs)[source]

Check if the content is allowed.

Parameters:

  • name (str) – The name of a specific evaluator.

  • **kwargs – Arbitrary keyword arguments to read args from.

datadiligence.bootstrap.list_evaluators()[source]

List the evaluators.

datadiligence.bootstrap.load_defaults(user_agent=None)[source]

Load the default evaluators.

datadiligence.bootstrap.register_evaluator(evaluator, name=None, overwrite=False)[source]

Register an evaluator.

Parameters:

  • evaluator (Evaluator) – The evaluator object.

  • name (str) – Key name of the evaluator.

  • overwrite (bool) – Whether or not to overwrite the evaluator if it already exists.

datadiligence.exceptions module

Exceptions for the package.

exception datadiligence.exceptions.DefaultEvaluatorNotFound(args)[source]

Bases: Exception

Raised when aa default evaluator can’t be determined.

exception datadiligence.exceptions.EvaluatorAlreadyRegistered(name)[source]

Bases: Exception

Raised when an evaluator is already registered.

exception datadiligence.exceptions.EvaluatorNotRegistered(name)[source]

Bases: Exception

Raised when an evaluator is not registered.

exception datadiligence.exceptions.HttpUnknownHeaderObject[source]

Bases: Exception

Raised when an HTTPRule is provided with an unknown header object.

exception datadiligence.exceptions.HttpUnknownResponseObject[source]

Bases: Exception

Raised when HTTPRule is provided with an unknown response object.

exception datadiligence.exceptions.NotEvaluator[source]

Bases: Exception

Raised when an object is not an evaluator.

exception datadiligence.exceptions.SpawningAIAPIError(message)[source]

Bases: Exception

Raised when the Spawning AI API returns an error.

exception datadiligence.exceptions.SpawningNoParam[source]

Bases: Exception

Raised when SpawningAPI isn’t provided with a list of urls.

exception datadiligence.exceptions.TDMRepNoParam[source]

Bases: Exception

Raised when TDMRepHeader isn’t provided with either an url, response, or headers object.

exception datadiligence.exceptions.XRobotsTagNoParam[source]

Bases: Exception

Raised when XRobotsTagHeader isn’t provided with either an url, response, or headers object.

datadiligence.utils module

Utility functions for package.

datadiligence.utils.get_url(url, user_agent=None)[source]

Get the URL and return the response object.

Parameters:

  • url (str) – The URL to get.

  • user_agent (str) – The user agent to use.

Returns:

The response object.

Return type:

requests.Response

Module contents

Respect generative AI opt-outs in your ML and training pipeline.