V0.1 (#115)

* setup v0.1 * rm \n in comments * QuickStart added for docs.scenarios * experiment result of citi bike update to 1-month version * V0.1 issues installation (#38) * replace pycurl to urllib * rm pycurl in test requirements * fix installation issue, update readme, speed up citibike Co-authored-by: Arthur Jiang <sjian@microsoft.com> * notebook link added for scenarios * fix: add missing licenses, refine long lines (#36) * scenario notebook update (#39) Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> * scenario doc modified according to comments * doc correction * scenario doc modified according to comments * remove examples * rl toolkit refactoring (#42) * rl toolkit refactoring * rename citi_bike * add new line Co-authored-by: Arthur Jiang <sjian@microsoft.com> * fix issue during building playground images (#44) * citibike doc modified according to comments * citibike doc update * scenario experiment result table changed * annotate the empty experiment results * merge master into v0.1 for later merging (#47) * setup v0.1 (#34) Co-authored-by: Arthur Jiang <sjian@microsoft.com> * fix pycurl install issue (#35) Co-authored-by: Arthur Jiang <ArthurSJiang@gmail.com> Co-authored-by: Arthur Jiang <sjian@microsoft.com> * refine readme * fix: recursive copy instead of plain copy (#45) * update maro overall image * typo * add key components * fix deployment issue in multi envs (#49) * typo * refine introduction * fix typo * fix readme * refine readme * refine readme * update readme docs link * fix ~/.maro not exist issue in build (#52) * fix deployment issue in multi envs * fix typo * fix ~/.maro not exist issue in build * skip deploy when build * update for comments * refine logo * fix toctree * fix & feat: invalid filename & maro grass status (#51) * fix: change invalid filename to valid * feat: add maro grass status * refine * refine * V0.1 test store update (#58) * updateed test_store.py * fixed unpicklable store bug Co-authored-by: ysqyang <v-yangqi@microsoft.com> * rename ecr 2 cim (#62) * correct dependencies * rename ecr to cim for hello, scenario, datalib and tests * remove unused dependency for build * add a new line at the end * a'd * V0.1 rename to cim in example (#63) * renamed ecr to cim and removed unwanted files * renamed ecr to cim in other places * fixed PR conflicts * merged with v0.1 Co-authored-by: ysqyang <v-yangqi@microsoft.com> * fix ut issue under window, now all runpass (#64) * correct dependencies * rename ecr to cim for hello, scenario, datalib and tests * install torch first before test dependencies * missing ecr 2 cim * V0.1 ecr rename (#65) * rename ecr to cim in docs * ecr renamed for notebooks * OOCL removed * rename ecr to cim * title level modified * notebook update Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> * temporarily disable weather, fix station id issue (#66) * fix deployment issue in multi envs * fix typo * fix ~/.maro not exist issue in build * skip deploy when build * update for comments * temporarily disable weather info * replace ecr with cim in setup.py * replace ecr in manifest * remove weather check when read data * fix station id issue * fix format * add TODO in comments * V0.1 ecr rename (#67) * rename ecr to cim in docs * ecr renamed for notebooks * OOCL removed * rename ecr to cim * title level modified * notebook update * topo renamed Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> * change the keys in metrics (#68) * correct dependencies * rename ecr to cim for hello, scenario, datalib and tests * install torch first before test dependencies * missing ecr 2 cim * rename metrics keys * remove transfer cost factor * rename cost to number * update setup * Update build_wheel.yml * Update build_wheel.yml * Update build_wheel.yml * Update build_wheel.yml * rm install from PyPi * V0.1 cim nb (#70) * added cim nb * fixed bugs in nb * fixed bugs in nb * fixed a bug in nb Co-authored-by: ysqyang <v-yangqi@microsoft.com> * the font size of doc diagrams is enlarged (#73) * the font size of doc diagrams is enlarged * disable APIDOC Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> * str() and repr() of DecisionEvent and Action added; notebook output updated (#74) Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> * Update deploy_gh_pages.yml * fix docker build * V0.1 feature docker ci (#77) * Update deploy_docker_image.yml * Update deploy_docker_image.yml * Update deploy_docker_image.yml * Update deploy_docker_image.yml * refine * add contributing Co-authored-by: Arthur Jiang <sjian@microsoft.com> * update playground docs * fixed a key error in K-step shaper for CIM (#80) Co-authored-by: ysqyang <v-yangqi@microsoft.com> * V0.1 baseline results (#81) * adding baseline results * updated ecr results * updated cim baseline results * added citibike no_action results * restored cim/hello.py * cim result update * citibike result update * added missing files * fixed a bug * updated citibike random repositioning results * completed baseline results * update citi bike results * format update * remove runner files Co-authored-by: ysqyang <v-yangqi@microsoft.com> Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> * duplicated notebook removed * update distributed example (#69) * update distributed example * add config.py for distributed mode examples * add new_line in config.py * repalce title with caps * add test and build badge (#76) * correct dependencies * rename ecr to cim for hello, scenario, datalib and tests * install torch first before test dependencies * missing ecr 2 cim * rename metrics keys * remove transfer cost factor * rename cost to number * add badge * change unittest to test * add docker image badge * remove unexist folders * ignore badges, ignore examples as module * remove __init__ file from examples * V0.1 k step key error fix (#84) * fixed a key error in K-step shaper for CIM * fixed key error in k-step shaper in dist_actor Co-authored-by: ysqyang <v-yangqi@microsoft.com> * fixed unpicklable bug in k step shaper (#87) Co-authored-by: ysqyang <v-yangqi@microsoft.com> * Update and rename unittests.yml to test.yml * code block format update * removed sentence above title (#90) Co-authored-by: ysqyang <v-yangqi@microsoft.com> * cim results update * 20 (#88) Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> * V0.1 import bug fix in example (#86) * fixed a key error in K-step shaper for CIM * fixing import bugs * removed a blank line and an unwanted import * added __init__ in components and changed import accordingly * added . before components * removed . before components * moved dist_actor and dist_learner out of components * fixed a bug * fixed import formatting * fixed import formatting * removed config read from single_process_launcher * restored absolute path for running dist_actor and dist_learner * fixed a bug Co-authored-by: ysqyang <v-yangqi@microsoft.com> * updated citibike resposition number column * fixed table formatiing * Update setup.py * fix pypi meta * refine doc string * fix classifier * fix pypi homepage * update badge * V0.1 bug cli (#71) * fix: previous fix cannot solve this problem * fix: loadBalancerSku is required in new azure-cli deployment * enable weather data (#96) * fix deployment issue in multi envs * fix typo * fix ~/.maro not exist issue in build * skip deploy when build * update for comments * temporarily disable weather info * replace ecr with cim in setup.py * replace ecr in manifest * remove weather check when read data * fix station id issue * fix format * add TODO in comments * add noaa weather source * fix weather reset and weather comment * add comment for weather data url * some format update * add badges for scenarios and platforms (#100) * correct dependencies * rename ecr to cim for hello, scenario, datalib and tests * install torch first before test dependencies * missing ecr 2 cim * rename metrics keys * remove transfer cost factor * rename cost to number * add badge * change unittest to test * add docker image badge * remove unexist folders * ignore badges, ignore examples as module * remove __init__ file from examples * add badges for scenario and platform * update badges * update static badges * fix * changed default logging levels (#98) * changed default logging levels * changed logger level back to DEBUG in Logger class Co-authored-by: ysqyang <v-yangqi@microsoft.com> * V0.1 citibike notebook bug (#94) * rewrote citibike nb using greedy agent * checked out missing files from v0.1 * refined notebooks * added output in greedy_policy.ipynb * added output * added output to cim nb * refined nb * renamed GreedyAgent to GreedyPolicy Co-authored-by: ysqyang <v-yangqi@microsoft.com> * update github workflow, add quality action (#111) * update github workflow, add quality action * update quality parameter * update quality * update quality parameter Co-authored-by: Arthur Jiang <sjian@microsoft.com> * delete proxy useless parameter (#114) Co-authored-by: Arthur Jiang <sjian@microsoft.com> Co-authored-by: Jinyu Wang <Wang.Jinyu@microsoft.com> Co-authored-by: Romic Huang <romic.hlc@outlook.com> Co-authored-by: Jinyu-W <53509467+Jinyu-W@users.noreply.github.com> Co-authored-by: kaiqli <59279714+kaiqli@users.noreply.github.com> Co-authored-by: Chaos Yu <chaos.you@gmail.com> Co-authored-by: Romic Huang <romic.kid@gmail.com> Co-authored-by: zhanyu wang <pocket_2001@163.com> Co-authored-by: ysqyang <ysqyang@gmail.com> Co-authored-by: ysqyang <v-yangqi@microsoft.com>
microsoft · Sep 28, 2020 · 594bdb3 · 594bdb3
1 parent 321bd61
commit 594bdb3
Show file tree

Hide file tree

Showing 19 changed files with 664 additions and 1,031 deletions.
diff --git a/.github/workflows/build_wheel.yml b/.github/workflows/build_wheel.yml
@@ -8,7 +8,7 @@ on:
   workflow_dispatch:
     inputs:
       logLevel:
-        description: 'package version'     
+        description: 'package version'
         required: true
         default: ''
 

diff --git a/.github/workflows/deploy_docker_image.yml b/.github/workflows/deploy_docker_image.yml
@@ -11,7 +11,7 @@ on:
   workflow_dispatch:
     inputs:
       logLevel:
-        description: 'package version'     
+        description: 'package version'
         required: true
         default: ''
 

diff --git a/.github/workflows/deploy_gh_pages.yml b/.github/workflows/deploy_gh_pages.yml
@@ -12,7 +12,7 @@ on:
   workflow_dispatch:
     inputs:
       logLevel:
-        description: 'package version'     
+        description: 'package version'
         required: true
         default: '0.1.1a0'
 

diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml
@@ -0,0 +1,23 @@
+name: quality
+
+on: [ push,  workflow_dispatch ]
+
+jobs:
+  check-quality:
+    runs-on: ubuntu-latest
+    name: A job to check my code quality
+    steps:
+    - name: Check code meets quality standards
+      id: code-inspector
+      uses: codeinspectorio/github-action@master
+      with:
+        repo_token: ${{ secrets.GITHUB_TOKEN }}
+        code_inspector_access_key: ${{ secrets.CODE_INSPECTOR_ACCESS_KEY }}
+        code_inspector_secret_key: ${{ secrets.CODE_INSPECTOR_SECRET_KEY }}
+        min_quality_grade: 'WARNING'
+        min_quality_score: '50'
+        max_defects_rate: '0.08'
+        max_complex_functions_rate: '0.08'
+        max_long_functions_rate: '0.08'
+        project_name: ''
+        max_timeout_sec: '600'
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -11,9 +11,9 @@ on:
       tags:
         description: 'Test scenario tags'
   push:
-    branches: [ master ]
+    branches: [ master, v0.1 ]
   pull_request:
-    branches: [ master ]
+    branches: [ master, v0.1 ]
 
 jobs:
   test:

diff --git a/examples/cim/dqn/single_process_launcher.py b/examples/cim/dqn/single_process_launcher.py
@@ -1,15 +1,14 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT license.
 
-import io
+
 import os
-import yaml
 
 import numpy as np
 
 from maro.simulator import Env
 from maro.rl import SimpleLearner, SimpleActor, AgentMode, KStepExperienceShaper, TwoPhaseLinearExplorer
-from maro.utils import Logger, convert_dottable
+from maro.utils import Logger
 
 from components.action_shaper import CIMActionShaper
 from components.agent_manager import DQNAgentManager
@@ -19,8 +18,12 @@
 
 
 if __name__ == "__main__":
+    # Step 1: initialize a CIM environment for using a toy dataset.
     env = Env(config.env.scenario, config.env.topology, durations=config.env.durations)
     agent_id_list = [str(agent_id) for agent_id in env.agent_idx_list]
+
+    # Step 2: create state, action and experience shapers. We also need to create an explorer here due to the
+    # greedy nature of the DQN algorithm.
     state_shaper = CIMStateShaper(**config.state_shaping)
     action_shaper = CIMActionShaper(action_space=list(np.linspace(-1.0, 1.0, config.agents.algorithm.num_actions)))
     if config.experience_shaping.type == "truncated":
@@ -34,15 +37,19 @@
                           "with_cache": config.exploration.with_cache
                           }
     explorer = TwoPhaseLinearExplorer(agent_id_list, config.general.total_training_episodes, **exploration_config)
+
+    # Step 3: create an agent manager.
     agent_manager = DQNAgentManager(name="cim_learner",
                                     mode=AgentMode.TRAIN_INFERENCE,
                                     agent_id_list=agent_id_list,
                                     state_shaper=state_shaper,
                                     action_shaper=action_shaper,
                                     experience_shaper=experience_shaper,
                                     explorer=explorer)
-    learner = SimpleLearner(trainable_agents=agent_manager,
-                            actor=SimpleActor(env=env, inference_agents=agent_manager),
+
+    # Step 4: Create an actor and a learner to start the training process.
+    actor = SimpleActor(env=env, inference_agents=agent_manager)
+    learner = SimpleLearner(trainable_agents=agent_manager, actor=actor,
                             logger=Logger("single_host_cim_learner", auto_timestamp=False))
 
     learner.train(total_episodes=config.general.total_training_episodes)

diff --git a/examples/citi_bike/greedy/launcher.py b/examples/citi_bike/greedy/launcher.py
@@ -13,7 +13,7 @@
     config = convert_dottable(raw_config)
 
 
-class GreedyAgent:
+class GreedyPolicy:
     def __init__(self, supply_top_k: int = 1, demand_top_k: int = 1):
         """
         Agent that executes a greedy policy. If the event type is supply, send as many bikes as possible to one of the
@@ -67,10 +67,10 @@ def choose_action(self, decision_event: DecisionEvent):
     if config.env.seed is not None:
         env.set_seed(config.env.seed)
 
-    agent = GreedyAgent(config.agent.supply_top_k, config.agent.demand_top_k)
+    policy = GreedyPolicy(config.agent.supply_top_k, config.agent.demand_top_k)
     metrics, decision_event, done = env.step(None)
     while not done:
-        metrics, decision_event, done = env.step(agent.choose_action(decision_event))
+        metrics, decision_event, done = env.step(policy.choose_action(decision_event))
 
     print(f"Greedy agent policy performance: {env.metrics}")
 

diff --git a/maro/cli/utils/params.py b/maro/cli/utils/params.py
@@ -4,10 +4,10 @@
 
 import logging
 
+
 class GlobalParams:
     PARALLELS = 5
-
-    LOG_LEVEL = logging.DEBUG
+    LOG_LEVEL = logging.INFO
 
 
 class GlobalPaths:

diff --git a/maro/communication/proxy.py b/maro/communication/proxy.py
@@ -6,9 +6,7 @@
 from enum import Enum
 import itertools
 import json
-import random
 import sys
-import string
 import signal
 import time
 from typing import List, Tuple, Dict, Union
@@ -31,7 +29,6 @@
 PORT = default_parameters.proxy.redis.port
 MAX_RETRIES = default_parameters.proxy.redis.max_retries
 BASE_RETRY_INTERVAL = default_parameters.proxy.redis.base_retry_interval
-SUFFIX_LENGTH = default_parameters.proxy.suffix_length
 FAULT_TOLERANT = default_parameters.proxy.fault_tolerant
 DELAY_FOR_SLOW_JOINER = default_parameters.proxy.delay_for_slow_joiner
 
@@ -52,15 +49,14 @@ class Proxy:
         redis_address (Tuple): Hostname and port of the Redis server,
         max_retries (int): Maximum number of retries before raising an exception,
         base_retry_interval (float = 0.1): The time interval between attempts,
-        suffix_length (int = 6): The suffix length,
         fault_tolerant (bool): Proxy can tolerate sending message error or not, default is False,
         log_enable (bool): Open logger or not, default is True.
     """
 
     def __init__(self, group_name: str, component_type: str, expected_peers: dict,
                  driver_type: DriverType = DriverType.ZMQ, driver_parameters: dict = None,
                  redis_address=(HOST, PORT), max_retries: int = MAX_RETRIES,
-                 base_retry_interval: float = BASE_RETRY_INTERVAL, suffix_length: int = SUFFIX_LENGTH,
+                 base_retry_interval: float = BASE_RETRY_INTERVAL,
                  fault_tolerant: bool = FAULT_TOLERANT, log_enable: bool = True):
         self._group_name = group_name
         self._component_type = component_type
@@ -71,9 +67,9 @@ def __init__(self, group_name: str, component_type: str, expected_peers: dict,
         self._driver_parameters = driver_parameters
         self._max_retries = max_retries
         self._retry_interval = base_retry_interval
-        self._suffix_length = suffix_length
         self._is_enable_fault_tolerant = fault_tolerant
         self._log_enable = log_enable
+        self._logger = InternalLogger(component_name=self._name) if self._log_enable else DummyLogger()
 
         try:
             self._redis_connection = redis.Redis(host=redis_address[0], port=redis_address[1])
@@ -87,8 +83,6 @@ def __init__(self, group_name: str, component_type: str, expected_peers: dict,
         self._onboard_peers_name_dict = {}
         self._message_cache = defaultdict(list)
 
-        self._logger = InternalLogger(component_name=self._name) if self._log_enable else DummyLogger()
-
         self._join()
 
     def _signal_handler(self, signum, frame):

diff --git a/maro/communication/utils/default_parameters.py b/maro/communication/utils/default_parameters.py
@@ -6,7 +6,6 @@
 
 
 proxy = convert_dottable({
-    "suffix_length": 6,
     "fault_tolerant": False,
     "delay_for_slow_joiner": 3,
     "redis": {

diff --git a/maro/utils/logger.py b/maro/utils/logger.py
@@ -106,7 +106,7 @@ class Logger:
     """
 
     def __init__(self, tag: str, format_: LogFormat = LogFormat.full, dump_folder: str = cwd, dump_mode: str = 'w',
-                 extension_name: str = 'log', auto_timestamp: bool = True, stdout_level="DEBUG"):
+                 extension_name: str = 'log', auto_timestamp: bool = True, stdout_level="INFO"):
         self._file_format = FORMAT_NAME_TO_FILE_FORMAT[format_]
         self._stdout_format = FORMAT_NAME_TO_STDOUT_FORMAT[format_] \
             if format_ in FORMAT_NAME_TO_STDOUT_FORMAT else \

diff --git a/notebooks/bike_repositioning/greedy_policy.ipynb b/notebooks/bike_repositioning/greedy_policy.ipynb
@@ -0,0 +1,118 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Greedy Policy\n",
+    "\n",
+    "In this notebook, we demonstrate using a simple greedy policy for [Citi Bike](https://maro.readthedocs.io/en/latest/scenarios/citi_bike.html), a real-world bike repositioning scenario. Our greedy policy is simple: if the event type is supply, the policy will make the current station send as many bikes as possible to one of k stations with the most empty docks. If the event type is demand, the policy will make the current station request as many bikes as possible from one of k stations with the most bikes. We use a heap data structure to find the top k supply/demand candidates from the action scope associated with each decision event."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import heapq\n",
+    "import random\n",
+    "\n",
+    "from maro.simulator import Env\n",
+    "from maro.simulator.scenarios.citi_bike.common import Action, DecisionEvent, DecisionType\n",
+    "\n",
+    "\n",
+    "class GreedyPolicy:\n",
+    "    def __init__(self, supply_top_k: int = 1, demand_top_k: int = 1):\n",
+    "        self._supply_top_k = supply_top_k\n",
+    "        self._demand_top_k = demand_top_k\n",
+    "\n",
+    "    def choose_action(self, decision_event: DecisionEvent):\n",
+    "        if decision_event.type == DecisionType.Supply:\n",
+    "            # Find k target stations with the most empty slots, randomly choose one of them and send as many bikes to\n",
+    "            # it as allowed by the action scope.\n",
+    "            top_k_demands = []\n",
+    "            for demand_candidate, available_docks in decision_event.action_scope.items():\n",
+    "                if demand_candidate == decision_event.station_idx:\n",
+    "                    continue\n",
+    "\n",
+    "                heapq.heappush(top_k_demands, (available_docks, demand_candidate))\n",
+    "                if len(top_k_demands) > self._demand_top_k:\n",
+    "                    heapq.heappop(top_k_demands)\n",
+    "\n",
+    "            max_reposition, target_station_idx = random.choice(top_k_demands)\n",
+    "            action = Action(decision_event.station_idx, target_station_idx, max_reposition)\n",
+    "        else:\n",
+    "            # Find k source stations with the most bikes, randomly choose one of them and request as many bikes from\n",
+    "            # it as allowed by the action scope.\n",
+    "            top_k_supplies = []\n",
+    "            for supply_candidate, available_bikes in decision_event.action_scope.items():\n",
+    "                if supply_candidate == decision_event.station_idx:\n",
+    "                    continue\n",
+    "\n",
+    "                heapq.heappush(top_k_supplies, (available_bikes, supply_candidate))\n",
+    "                if len(top_k_supplies) > self._supply_top_k:\n",
+    "                    heapq.heappop(top_k_supplies)\n",
+    "\n",
+    "            max_reposition, source_idx = random.choice(top_k_supplies)\n",
+    "            action = Action(source_idx, decision_event.station_idx, max_reposition)\n",
+    "\n",
+    "        return action"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Interaction with the Greedy Policy\n",
+    "\n",
+    "This environment is driven by [real trip history data](https://s3.amazonaws.com/tripdata/index.html) from Citi Bike. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Greedy policy performance: {'trip_requirements': 17729, 'bike_shortage': 12305, 'operation_number': 229229}\n"
+     ]
+    }
+   ],
+   "source": [
+    "env = Env(scenario=\"citi_bike\", topology=\"ny.201801\", start_tick=0, durations=2880, snapshot_resolution=10)\n",
+    "policy = GreedyPolicy()\n",
+    "metrics, decision_event, done = env.step(None)\n",
+    "while not done:\n",
+    "    metrics, decision_event, done = env.step(policy.choose_action(decision_event))\n",
+    "\n",
+    "print(f\"Greedy policy performance: {env.metrics}\")\n",
+    "env.reset()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}