Skip to content

Latest commit

 

History

History
113 lines (86 loc) · 6.43 KB

README.md

File metadata and controls

113 lines (86 loc) · 6.43 KB
Raw

HTTP logs workload

This workload is based on Web server logs from the 1998 Football world cup.

Modifications:

  • Applied number to IP conversion as suggested in the original readme
  • Removed illegal characters in "object_mappings.sort"
  • Transformed the source data to a bulk-friendly JSON format (ignoring all entries that contained unrecognised / problematic characters and invalid IP addresses like "0"; around 0.001% of the source data was lost due to this approach)

Example Document

{
  "@timestamp": 898459201,
  "clientip": "211.11.9.0",
  "request": "GET /english/index.html HTTP/1.0",
  "status": 304,
  "size": 0
}

Alternatively, an unparsed set of documents are also provided. The unparsed data set is identical to the standard data set, except the timestamp is ISO8601 and all the fields are unparsed via the message field. For example:

{"message" : "211.11.9.0 - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}

Parameters

This workload allows the following parameters to be specified using --workload-params:

  • bulk_size (default: 5000)
  • bulk_indexing_clients (default: 8): Number of clients that issue bulk indexing requests.
  • ingest_percentage (default: 100): A number between 0 and 100 that defines how much of the document corpus should be ingested.
  • conflicts (default: "random"): Type of id conflicts to simulate. Valid values are: 'sequential' (A document id is replaced with a document id with a sequentially increasing id), 'random' (A document id is replaced with a document id with a random other id).
  • conflict_probability (default: 25): A number between 0 and 100 that defines the probability of id conflicts. This requires to run the respective test_procedure. Combining conflicts=sequential and conflict-probability=0 makes Benchmark generate index ids by itself, instead of relying on OpenSearch's automatic id generation.
  • number_of_replicas (default: 0)
  • number_of_shards (default: 5)
  • query_cache_enabled (default: false)
  • requests_cache_enabled (default: false)
  • source_enabled (default: true): A boolean defining whether the _source field is stored in the index.
  • index_settings: A list of index settings. Index settings defined elsewhere (e.g. number_of_replicas) need to be overridden explicitly.
  • cluster_health (default: "green"): The minimum required cluster health.
  • ingest_pipeline: Only applicable for --test-procedure=append-index-only-with-ingest-pipeline, selects which ingest node pipeline to run. Valid options are 'baseline' (default), 'grok' and 'geoip'. For example: --test-procedure=append-index-only-with-ingest-pipeline --workload-params="ingest_pipeline:'baseline'"
  • error_level (default: "non-fatal"): Available for bulk operations only to specify ignore-response-error-level.
  • target_throughput (default: default values for each operation): Number of requests per second, none for no limit.
  • search_clients: Number of clients that issues search requests.

Beta Feature: Increasing the size of the data corpus

This workload provides for a feature to use a generated data corpus in lieu of the provided corpora files (which currently total ~31 GB.) The generated corpus could, for instance, be 100 GB or more. For more details on generating such a corpus, run the following command:

  expand-data-corpus.py -h

Once a corpus has been generated, it can be used for a test by supplying the following parameter via --workoad-params:

  • generated_corpus:t: Use the generated data corpus instead of the corpora files packaged with this track

If there are multiple generated corpora files, they are all used concurrently. Ingestion of the generated and the default corpora are mutually exclusive in any single OSB run. Once ingested, however, queries packaged with this workload will operate on the entire loaded data set.

Keep in mind when running expand-data-corpus.py, the feature by default influences the ~/.benchmark/benchmarks/workloads/default/http_logs and ~/.benchmark/benchmarks/data/ directories. To choose a different workload repository other than default, supply a different workload repository with -r parameter.

License

Original license text:

           Copyright (C) 1997, 1998, 1999 Hewlett-Packard Company
                         ALL RIGHTS RESERVED.

  The enclosed software and documentation includes copyrighted works
  of Hewlett-Packard Co. For as long as you comply with the following
  limitations, you are hereby authorized to (i) use, reproduce, and
  modify the software and documentation, and to (ii) distribute the
  software and documentation, including modifications, for
  non-commercial purposes only.

  1.  The enclosed software and documentation is made available at no
      charge in order to advance the general development of
      the Internet, the World-Wide Web, and Electronic Commerce.

  2.  You may not delete any copyright notices contained in the
      software or documentation. All hard copies, and copies in
      source code or object code form, of the software or
      documentation (including modifications) must contain at least
      one of the copyright notices.

  3.  The enclosed software and documentation has not been subjected
      to testing and quality control and is not a Hewlett-Packard Co.
      product. At a future time, Hewlett-Packard Co. may or may not
      offer a version of the software and documentation as a product.

  4.  THE SOFTWARE AND DOCUMENTATION IS PROVIDED "AS IS".
      HEWLETT-PACKARD COMPANY DOES NOT WARRANT THAT THE USE,
      REPRODUCTION, MODIFICATION OR DISTRIBUTION OF THE SOFTWARE OR
      DOCUMENTATION WILL NOT INFRINGE A THIRD PARTY'S INTELLECTUAL
      PROPERTY RIGHTS. HP DOES NOT WARRANT THAT THE SOFTWARE OR
      DOCUMENTATION IS ERROR FREE. HP DISCLAIMS ALL WARRANTIES,
      EXPRESS AND IMPLIED, WITH REGARD TO THE SOFTWARE AND THE
      DOCUMENTATION. HP SPECIFICALLY DISCLAIMS ALL WARRANTIES OF
      MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

  5.  HEWLETT-PACKARD COMPANY WILL NOT IN ANY EVENT BE LIABLE FOR ANY
      DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES
      (INCLUDING LOST PROFITS) RELATED TO ANY USE, REPRODUCTION,
      MODIFICATION, OR DISTRIBUTION OF THE SOFTWARE OR DOCUMENTATION.