[openstack-dev] [MagnetoDB] Bulk load API draft

Illia Khudoshyn ikhudoshyn at mirantis.com
Wed May 28 10:52:34 UTC 2014


Hi Ilya

As for 'string' vs 'number', in MDB REST API we pass numers as strings
since we want to support big ints, so I just wanted to be conform.

As for the last parameter name, I'd prefer 'failed_items', 'cos we 1)
already have 'failed' and I think it would be good if they match 2) they
were actually processed but failed


On Wed, May 28, 2014 at 1:03 PM, Ilya Sviridov <isviridov at mirantis.com>wrote:

> Hello Illia,
>
> Great job!
>
> Several moments about spec itself.
>
> "read": "string",
> "processed": "string",
> "failed": "string",
>
> I believe all of them are integers, but not strings.
>
> About Dmitri's suggestion to rename "errors", I would also prefer to do it
> as far as error itself goes below with item.
> What do you think about  "unprocessed_items"?
>
> And I think we can go with it.
>
> Ilya
>
>
>
> On Wed, May 28, 2014 at 10:28 AM, Illia Khudoshyn <ikhudoshyn at mirantis.com
> > wrote:
>
>> Hi Dima,
>>
>> Sounds good, thank you for the point.
>>
>>
>> On Tue, May 27, 2014 at 7:34 PM, Dmitriy Ukhlov <dukhlov at mirantis.com>wrote:
>>
>>> Hi Illia,
>>>
>>> Looks good. But I suggest to return all of these fields for positive
>>> request as well as for error request:
>>>
>>>     "read": "string",
>>>     "processed": "string",
>>>     "failed": "string",
>>>
>>> but leave next fields optional and fill them in case of error response
>>> ("failed" > 0) to specify what exactly was happened:
>>>
>>> "last_read":
>>> "errors" (maybe not processed will be better)
>>>
>>>
>>>
>>> On Tue, May 27, 2014 at 3:39 PM, Illia Khudoshyn <
>>> ikhudoshyn at mirantis.com> wrote:
>>>
>>>> Hi openstackers,
>>>>
>>>> While working on bulk load, I found previously proposed batch-oriented
>>>> asynchronous approach both resource consuming on server side and somewhat
>>>> complicated to use.
>>>> So I tried to outline some more straightforward streaming way of
>>>> uploading data.
>>>>
>>>> By the link below you can found a draft for a new streaming API
>>>> https://wiki.openstack.org/wiki/MagnetoDB/streamingbulkload.
>>>>
>>>> Any feedback is welcome as usual.
>>>>
>>>>
>>>>
>>>> On Wed, May 14, 2014 at 5:04 PM, Illia Khudoshyn <
>>>> ikhudoshyn at mirantis.com> wrote:
>>>>
>>>>> Hi openstackers,
>>>>>
>>>>> I'm working on bulk load for MagnetoDB, the facility for inserting
>>>>> large amounts of data, like,  millions of rows, gigabytes of data. Below is
>>>>> the link to draft API description.
>>>>>
>>>>>
>>>>> https://wiki.openstack.org/wiki/MagnetoDB/bulkload#.5BDraft.5D_MagnetoDB_Bulk_Load_workflow_and_API
>>>>>
>>>>> Any feedback is welcome.
>>>>>
>>>>> --
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Illia Khudoshyn,
>>>>> Software Engineer, Mirantis, Inc.
>>>>>
>>>>>
>>>>>
>>>>> 38, Lenina ave. Kharkov, Ukraine
>>>>>
>>>>> www.mirantis.com <http://www.mirantis.ru/>
>>>>>
>>>>> www.mirantis.ru
>>>>>
>>>>>
>>>>>
>>>>> Skype: gluke_work
>>>>>
>>>>> ikhudoshyn at mirantis.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Best regards,
>>>>
>>>> Illia Khudoshyn,
>>>> Software Engineer, Mirantis, Inc.
>>>>
>>>>
>>>>
>>>> 38, Lenina ave. Kharkov, Ukraine
>>>>
>>>> www.mirantis.com <http://www.mirantis.ru/>
>>>>
>>>> www.mirantis.ru
>>>>
>>>>
>>>>
>>>> Skype: gluke_work
>>>>
>>>> ikhudoshyn at mirantis.com
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Dmitriy Ukhlov
>>> Mirantis Inc.
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>>
>> Best regards,
>>
>> Illia Khudoshyn,
>> Software Engineer, Mirantis, Inc.
>>
>>
>>
>> 38, Lenina ave. Kharkov, Ukraine
>>
>> www.mirantis.com <http://www.mirantis.ru/>
>>
>> www.mirantis.ru
>>
>>
>>
>> Skype: gluke_work
>>
>> ikhudoshyn at mirantis.com
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Best regards,

Illia Khudoshyn,
Software Engineer, Mirantis, Inc.



38, Lenina ave. Kharkov, Ukraine

www.mirantis.com <http://www.mirantis.ru/>

www.mirantis.ru



Skype: gluke_work

ikhudoshyn at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140528/ab2f182c/attachment.html>


More information about the OpenStack-dev mailing list