Jump to content
XPEnology Community

Develop and refine the DS3622xs+ loader


yanjun

Recommended Posts

This is nuts!

Why does the ixgbe extension keep trying to download and install itself multiple times, only to fail and kill the build?

 

EDIT: broadwellnk-7.0.1-42218 DS3622xs+

 

I'm using the 4.5 version of the img file and ran ./rploader update now to grab all the newest scripts.

 

The extension file is in the default build now, right?

 

Can I enter something in the user_config.json file or elsewhere to manually exclude this from trying to load?

 

Thanks

Link to comment
Share on other sites

1 minute ago, tlsnine said:

This is nuts!

Why does the ixgbe extension keep trying to download and install itself multiple times, only to fail and kill the build?

 

EDIT: broadwellnk-7.0.1-42218 DS3622xs+

 

I'm using the 4.5 version of the img file and ran ./rploader update now to grab all the newest scripts.

 

The extension file is in the default build now, right?

 

Can I enter something in the user_config.json file or elsewhere to manually exclude this from trying to load?

 

Thanks

IXGBE is native in Loader, you don't have to add it.

Link to comment
Share on other sites

20 minutes ago, tlsnine said:

But the tinycore build script keeps trying to grab it remotely and add multiple instances.

How do I keep it from doing that?

 

True.. that has proved to fit most users. You just happen to be the exclusion. You might edit rploader.sh down at line #1158 and comment listmodules line.  Just insert a # or delete the line 

 

Anyway i fixed the missing reference.

Edited by pocopico
  • Like 2
Link to comment
Share on other sites

It was to play around, as i found this P222 in mY Hardware stuff

But i am not happy with it, will switch back to onboard controller.
Maybe satamap are wrong, but i am not able to create volumes after i add more than 2 Disks
They stay on status Dedected.

Not sure what is better hardware raid or let DSM manage it.

Link to comment
Share on other sites

2 minutes ago, altas said:

It was to play around, as i found this P222 in mY Hardware stuff

But i am not happy with it, will switch back to onboard controller.
Maybe satamap are wrong, but i am not able to create volumes after i add more than 2 Disks
They stay on status Dedected.

Not sure what is better hardware raid or let DSM manage it.

Definitely let DSM manage it. There is a lot of argument re: software raid or hardware raid but imo software raid is far better now due to large disk sizes and file system advancements.

Link to comment
Share on other sites

Why is tinycore downloading the dsm .pat file from this China site?

 

[#] Updating extensions... [OK]
[#] PAT file /home/tc/redpill-load/cache/ds3617xs_42218.pat not found - downloading from https://cndl.synology.cn/download/DSM/release/7.0.1/42218/DSM_DS3617xs_42218.pat

 

The site is often down and cannot download it, then you get an error message

I have tried it with DS3622xs+ and DS3617xs, both time gives time out and install stops.

 

Where can you change the download directory to https://global.download.synology.com/download/DSM/release/7.0.1/42218/DSM_DS3622xs+_42218.pat

 

It worked once the download, but now I am trying to download and install again and keeps failing.

Thank you

 

Kind regards

Link to comment
Share on other sites

14 minutes ago, altas said:

strange.. i never had any problem to downlod from cndl.. i did it right now.

why not pre-download and but it in the cach Folder ?

 

Yesterday I had no issues, could download it easy.

But today, it will not download, it does not matter which version.

I can download it from the global site without any issues.

 

So maybe using the global sit makes it easier ?

 

If I put it in the cache folder, the script failed:

 

Quote

Starting loader creation
Checking user_config.json : Done
Cache directory OK
sudo: ./build-loader.sh: command not found
FAILED : Loader creation failed check the output for any errors

 

Edited by MSXGames
Link to comment
Share on other sites

@pocopico @yanjun

I'm trying to install DS3622xs+ on ESXi with SAS2308 card, and I have an issue.

 

While configured with VMXNET3 card, even with ext installed, no IP, switching to E1000e I get an IP.

 

But each time I launch the install, it goes to 100%, reboot, then get back to "ready to install"

 

SynologyNAS login: [  115.251161] md1: detected capacity change from 2147418112 to 0
[  115.252243] md: md1: set sdb2 to auto_remap [0]
[  115.253022] md: md1 stopped.
[  115.253531] md: unbind<sdb2>
[  115.259305] md: export_rdev(sdb2)
[  118.527132] md: bind<sdb1>
[  118.528009] md/raid1:md0: active with 1 out of 16 mirrors
[  118.536635] md0: detected capacity change from 0 to 2549940224
[  121.574549] md: bind<sdb2>
[  121.575483] md/raid1:md1: active with 1 out of 16 mirrors
[  121.584067] md1: detected capacity change from 0 to 2147418112
[  122.167089] EXT4-fs (md0): couldn't mount as ext3 due to feature incompatibilities
[  122.200654] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
[  122.472271] <redpill/pmu_shim.c:310> Got 1 bytes from PMU: reason=1 hex={2d} ascii="-"
[  122.622300] <redpill/pmu_shim.c:310> Got 1 bytes from PMU: reason=1 hex={72} ascii="r"
[  122.624060] <redpill/pmu_shim.c:239> Executing cmd OUT_SCHED_UP_OFF handler cmd_shim_noop+0x0/0x30 [redpill]
[  122.626298] <redpill/pmu_shim.c:45> vPMU received OUT_SCHED_UP_OFF using 1 bytes - NOOP
[  122.630122] EXT4-fs (md0): couldn't mount as ext3 due to feature incompatibilities
[  122.662250] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
[  122.765976] EXT4-fs (md0): couldn't mount as ext3 due to feature incompatibilities
[  122.772601] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
[  130.121550] ext2: synoboot2 mounted, process=updater
[  130.124459] vfat: synoboot2 mounted, process=updater
[  130.129458] synoboot2 unmounted, process=updater
[  130.137883] ext2: synoboot2 mounted, process=updater
[  130.138958] vfat: synoboot2 mounted, process=updater
[  130.143760] synoboot2 unmounted, process=updater
[  130.154804] ext2: synoboot2 mounted, process=updater
[  130.155865] vfat: synoboot2 mounted, process=updater
[  130.160829] synoboot2 unmounted, process=updater
[  130.171098] ext2: synoboot2 mounted, process=updater
[  130.172135] vfat: synoboot2 mounted, process=updater
[  132.720588] synoboot2 unmounted, process=updater
[  137.732840] ext2: synoboot2 mounted, process=updater
[  137.733957] vfat: synoboot2 mounted, process=updater
[  137.786905] synoboot2 unmounted, process=updater
[  137.798075] ext2: synoboot1 mounted, process=updater
[  137.799176] vfat: synoboot1 mounted, process=updater
[  137.805167] synoboot1 unmounted, process=updater
[  137.818233] <redpill/intercept_execve.c:82> Blocked ./H2OFFT-Lx64 from running
[  137.819878] Module [phy_alloc] is removed.

 

Edit : Maybe the Dogodefi's repo currently broken with modifications (FS6400) ?

 

Because taking last action build for DS3622xs+ here works :

https://github.com/dogodefi/redpill-loader-action/actions/runs/1865605264

 

image.thumb.png.6cdd1629e991c01119845d07279f3e5e.png

 

Oups, in english will be better :

SO currently, my test disk 4Tb works well with the LSI SAS2308 HBA card in passthrough on the ESXi.

Still the little issue with LSI disks starting at disk 2, no matter the sata settings.

image.thumb.png.ed95c6649e62ac45e346c4096514d1da.png

 

So there is no issue with disks >2 Tb, it is a false positive in loader DSM install mode.

 

 

DS3622xs+.log

Edited by Orphée
Link to comment
Share on other sites

On 2/18/2022 at 8:16 PM, cferra said:

Looks like pocopicos repo has mlx drivers for 22xs. Awesome

as 3615/17 also 3622 comes with mlx4/5 in 7.0 its version 4.1-1.0.2

the one in the repository is 3.0-1 (i guess that the one from kernel 4.4.180)

3617 had newer drivers the kernel and 3615 so 3622 might have even newer drivers

beside checking kernel modules for version numbers its also possible to look on synologys homepage what nic's are supported for 3617 or 3622

 

 

 

  • Like 1
Link to comment
Share on other sites

 

 

CPU : i3-4160T
M/B : DT-H81DL GAIDA
LAN : RealTek r8111F DUAL
RAID : Dell Perc H310 , IT Mode
LOADER : Tinycore 0.4.5.1 Native Lastest Update
EXT : r8169, mpt3sas (yesterday Lastest Update)
DSM : DS3622xs,+ DSM 7.0.1

HDD : 4ea Sata Drives

 

To @pocopico

 

Like the attached screenshot, the entire health information, which is the details of S.M.A.R.T., is not visible. 
One of them, the bad sector count, seems to be marked negative in the bad sector count of the HDD/SDD main information because there is no information.
Can you also check this issue?
What log is needed to supplement this part?
I am very happy that the stabilization of DS3622xs+ using the HBA interface card is being completed first.
I am always grateful for your hard work.

스크린샷 2022-02-20 오전 10.01.42.png

Link to comment
Share on other sites

With mpt3sas using a 9305-24i even on jun’s loader I’m not seeing smart data there at all - check this below, is this an indication of a driver issue or a drive issue?
8c99b39a9f8994f2a83dc9a9cd478f6c.png

It does seem to show properly on another server containing a 9201-16i assuming using the mpt2sas driver.

fa251527412965eff860447a1be7ee88.png


Sent from my iPhone using Tapatalk

Link to comment
Share on other sites

@pocopico

 

The Dell H310 6Gbps SAS HBA (LSI 9211-8i P20 IT Mode Firmware Update) seems to require not only mpt3sas drivers but also mpt2sas drivers.
Even referring to the driver loading log of Jun loader 1.04b for DS918+, it was confirmed that both were utilized.
So, I tried to add the mpt2sas driver manually,
An error occurs at the final loader build stage as follows.
Can I ask you to check this part?

 

 

( Jun Loader DSM 6.2.3 DS918+)

root@NAS4:~# dmesg | grep mpt

[    0.000000] Preemptible hierarchical RCU implementation.

[    2.757193] mptctl: Registered with Fusion MPT base driver

[    2.757195] mptctl: /dev/mptctl @ (major,minor=10,220)

[    2.779415] mpt3sas version 09.102.00.00 loaded

[    2.779544] mpt3sas 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control

[    2.781591] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8006160 kB)

[    2.836673] mpt2sas_cm0: MSI-X vectors supported: 1, no of cores: 4, max_msix_vectors: -1

[    2.836721] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 27

[    2.836723] mpt2sas_cm0: iomem(0x00000000f7940000), mapped(0xffffc90000260000), size(65536)

[    2.836724] mpt2sas_cm0: ioport(0x000000000000e000), size(256)

[    2.930127] mpt2sas_cm0: Allocated physical memory: size(7445 kB)

[    2.930129] mpt2sas_cm0: Current Controller Queue Depth(3307),Max Controller Queue Depth(3432)

[    2.930130] mpt2sas_cm0: Scatter Gather Elements per IO(128)

[    2.975727] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03), BiosVersion(07.39.02.00)

[    2.975729] mpt2sas_cm0: Protocol=(

[    2.977740] mpt2sas_cm0: sending port enable !!

[    3.126435] Got empty serial number. Generate serial number from product.

[    4.502776] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x), phys(8)

[   10.380271] mpt2sas_cm0: port enable: SUCCESS

[   76.371203] Got empty serial number. Generate serial number from product.

[   76.381080] Got empty serial number. Generate serial number from product.

 

==================================================================================

 

CPU : i3-4160T
M/B : DT-H81DL GAIDA
LAN : RealTek r8111F DUAL
RAID : Dell Perc H310 , IT Mode
LOADER : Tinycore 0.4.5.1 Native Lastest Update
EXT : r8169, mpt3sas (yesterday Lastest Update)
DSM : DS3622xs,+ DSM 7.0.1

HDD : 4ea Sata Drives

 

 

root@box:/home/tc# ./rploader.sh ext broadwellnk-7.0.1-42218 add https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

 

jq: error (at <stdin>:1): Cannot iterate over null (null)

jq: error (at <stdin>:1): Cannot iterate over null (null)

Loader source : https://github.com/dogodefi/redpill-load.git Loader Branch : develop

Redpill module source : https://github.com/dogodefi/redpill-lkm.git : Redpill module branch : develop

Extensions :

Extensions URL :

TOOLKIT_URL : https://sourceforge.net/projects/dsgpl/files/toolkit/DSM7.0/ds.broadwellnk-7.0.dev.txz/download

TOOLKIT_SHA : 0d9edca67d9e7e14c2529bbb58341b623936124d5264f71f1e4acbacf3ea202d

SYNOKERNEL_URL : https://sourceforge.net/projects/dsgpl/files/Synology NAS GPL Source/25426branch/broadwellnk-source/linux-4.4.x.txz/download

SYNOKERNEL_SHA : d3e85eb80f16a83244fcae6016ab6783cd8ac55e3af2b4240455261396e1e1be

COMPILE_METHOD : toolkit_dev

TARGET_PLATFORM       : broadwellnk

TARGET_VERSION    : 7.0.1

TARGET_REVISION : 42218

REDPILL_LKM_MAKE_TARGET : dev-v7

KERNEL_MAJOR : 4

MODULE_ALIAS_FILE= modules.alias.4.json

Checking Internet Access -> OK

Cloning into 'redpill-lkm'...

remote: Enumerating objects: 1142, done.

remote: Counting objects: 100% (1142/1142), done.

remote: Compressing objects: 100% (588/588), done.

remote: Total 1142 (delta 792), reused 869 (delta 547), pack-reused 0

Receiving objects: 100% (1142/1142), 464.83 KiB | 2.19 MiB/s, done.

Resolving deltas: 100% (792/792), done.

Cloning into 'redpill-load'...

remote: Enumerating objects: 867, done.

remote: Counting objects: 100% (334/334), done.

remote: Compressing objects: 100% (256/256), done.

remote: Total 867 (delta 168), reused 98 (delta 41), pack-reused 533

Receiving objects: 100% (867/867), 24.36 MiB | 8.80 MiB/s, done.

Resolving deltas: 100% (374/374), done.

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

[OK]

[#] ========================================== pocopico.mpt2sas ==========================================

[#] Extension name: mpt2sas

[#] Description: Adds Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt2sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

root@box:/home/tc# ./rploader.sh build broadwellnk-7.0.1-42218

jq: error (at <stdin>:1): Cannot iterate over null (null)

jq: error (at <stdin>:1): Cannot iterate over null (null)

Loader source : https://github.com/dogodefi/redpill-load.git Loader Branch : develop

Redpill module source : https://github.com/dogodefi/redpill-lkm.git : Redpill module branch : develop

Extensions :

Extensions URL :

TOOLKIT_URL : https://sourceforge.net/projects/dsgpl/files/toolkit/DSM7.0/ds.broadwellnk-7.0.dev.txz/download

TOOLKIT_SHA : 0d9edca67d9e7e14c2529bbb58341b623936124d5264f71f1e4acbacf3ea202d

SYNOKERNEL_URL : https://sourceforge.net/projects/dsgpl/files/Synology NAS GPL Source/25426branch/broadwellnk-source/linux-4.4.x.txz/download

SYNOKERNEL_SHA : d3e85eb80f16a83244fcae6016ab6783cd8ac55e3af2b4240455261396e1e1be

COMPILE_METHOD : toolkit_dev

TARGET_PLATFORM       : broadwellnk

TARGET_VERSION    : 7.0.1

TARGET_REVISION : 42218

REDPILL_LKM_MAKE_TARGET : dev-v7

KERNEL_MAJOR : 4

MODULE_ALIAS_FILE= modules.alias.4.json

Checking Internet Access -> OK

Checking if a newer version exists on the repo -> Version is current

Redpill sources already downloaded, pulling latest

Already up to date.

Loader sources already downloaded, pulling latest

Already up to date.

Using static compiled redpill extension

Looking for redpill for : ds3622xsp_42218

Getting file https://raw.githubusercontent.com/pocopico/rp-ext/master/redpill/releases/redpill-4.4.180plus-broadwellnk.tgz

Extracting module

Getting file https://raw.githubusercontent.com/pocopico/rp-ext/master/redpill/src/check-redpill.sh

Got redpill-linux-v4.4.180+.ko

Testing modules.alias.4.json -> File OK

------------------------------------------------------------------------------------------------

It looks that you will need the following modules :

 

 

Found VGA Controller : pciid 8086d0000041e  Required Extension : i915

Searching for matching extension for i915

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

######################################################################## 100.0%

[OK]

[#] ========================================== pocopico.mpt3sas ==========================================

[#] Extension name: mpt3sas

[#] Description: Adds LSI MPT Fusion SAS 3.0 Device Driver Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt3sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3622xsp_42218 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

Found SAS Controller : pciid 1000d00000072  Required Extension : mpt3sas

Searching for matching extension for mpt3sas

Found matching extension :

"https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json"

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

######################################################################## 100.0%

[OK]

[#] ========================================== pocopico.r8169 ==========================================

[#] Extension name: r8169

[#] Description: Adds RealTek RTL-8169 Gigabit Ethernet driver Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/r8169

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3622xsp_42218 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

Found Ethernet Interface : pciid 10ecd00008168 Required Extension : r8169

Searching for matching extension for r8169

Found matching extension :

"https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json"

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

######################################################################## 100.0%

 

[!] Extension is already added (index exists at /home/tc/redpill-load/custom/extensions/pocopico.r8169/pocopico.r8169.json). For more info use "ext-manager.sh info pocopico.r8169"

 

*** Process will exit ***

Found Ethernet Interface : pciid 10ecd00008168 Required Extension : r8169

Searching for matching extension for r8169

Found matching extension :

"https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json"

------------------------------------------------------------------------------------------------

Starting loader creation

Checking user_config.json : Done

Cache directory OK

[#] Checking runtime for required tools... [OK]

[#] Updating extensions...

 

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

[OK]

[#] ========================================== dogodefi.mpt3sas ==========================================

[#] Extension name: mpt3sas

[#] Description: Adds LSI MPT Fusion SAS 3.0 Device Driver Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/dogodefi/mpt3sas

[#] Software author: https://github.com/dogodefi

[#] Update URL: https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json

[#] Platforms supported: fs6400_42218 ds3622xsp_42218 ds918p_42218 rs3617xsp_42218

[#] =======================================================================================

 

[#] Checking runtime for required tools... [OK]

[#] Updating dogodefi.mpt3sas extension...

[#] Downloading remote file https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension dogodefi.mpt3sas index is already up to date

[#] Updating pocopico.mpt2sas extension...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension pocopico.mpt2sas index is already up to date

[#] Updating pocopico.mpt3sas extension...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension pocopico.mpt3sas index is already up to date

[#] Updating pocopico.r8169 extension...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension pocopico.r8169 index is already up to date

[#] Updating pocopico.r8169 extension... [OK]

[#] Checking runtime for required tools... [OK]

[#] Updating ds3622xsp_42218 platforms extensions...

[#] Downloading remote file https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/releases/ds3622xsp_42218.json to /home/tc/redpill-load/custom/extensions/_ext_new_rcp.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Filling-in newly downloaded recipe for extension dogodefi.mpt3sas platform ds3622xsp_42218

[#] Downloading remote file https://github.com/dogodefi/mpt3sas/raw/offical/releases/mpt3sas-4.4.180plus-broadwellnk.tgz to /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/mpt3sas-4.4.180plus-broadwellnk.tgz

############################################################################################################################################################################################################## 100.0%############################################################################################################################################################################################################## 100.0%

[#] Verifying /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/mpt3sas-4.4.180plus-broadwellnk.tgz file... [OK]

[#] Unpacking files from /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/mpt3sas-4.4.180plus-broadwellnk.tgz to /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/... [OK]

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/src/check-mpt3sas.sh to /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/check-mpt3sas.sh

############################################################################################################################################################################################################## 100.0%

[#] Verifying /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/check-mpt3sas.sh file... [OK]

[#] Successfully processed recipe for extension dogodefi.mpt3sas platform ds3622xsp_42218

[-] The extension pocopico.mpt2sas was found. However, the extension index has no recipe for ds3622xsp_42218 platform. It may not be

[-] supported on that platform, or author didn't updated it for that platform yet. You can try running

[-] "ext-manager.sh update" to refresh indexes for all extensions manually. Below are the currently known information about

[-] the extension stored locally:

[#] ========================================== pocopico.mpt2sas ==========================================

[#] Extension name: mpt2sas

[#] Description: Adds Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt2sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

[-] Failed to update recipe for pocopico.mpt2sas extension for platform ds3622xsp_42218. The script will terminate as you do not

[-] have previously downloaded recipe which can be used if download fails. Try again later. If problem

[-] persists contact the extension packer for support (displayed below)

[#] ========================================== pocopico.mpt2sas ==========================================

[#] Extension name: mpt2sas

[#] Description: Adds Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt2sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

[!] Cannot continue due to previous errors (see above)

 

*** Process will exit ***

[!] Failed to update all extensions for ds3622xsp_42218 platform - see errors above

 

*** Process will exit ***

FAILED : Loader creation failed check the output for any errors

root@box:/home/tc#

Edited by Peter Suh
Link to comment
Share on other sites

6 hours ago, IG-88 said:

as 3615/17 also 3622 comes with mlx4/5 in 7.0 its version 4.1-1.0.2

the one in the repository is 3.0-1 (i guess that the one from kernel 4.4.180)

3617 had newer drivers the kernel and 3615 so 3622 might have even newer drivers

beside checking kernel modules for version numbers its also possible to look on synologys homepage what nic's are supported for 3617 or 3622

 

 

 

True, but tc downloads and installs the extensions anyway for these models, maybe @pocopico could add logic to the build process that if the model has a network card that is included by default to skip the addition of the rp-ext for that device to the build. 

Link to comment
Share on other sites

5 hours ago, Peter Suh said:

@pocopico

 

The Dell H310 6Gbps SAS HBA (LSI 9211-8i P20 IT Mode Firmware Update) seems to require not only mpt3sas drivers but also mpt2sas drivers.
Even referring to the driver loading log of Jun loader 1.04b for DS918+, it was confirmed that both were utilized.
So, I tried to add the mpt2sas driver manually,
An error occurs at the final loader build stage as follows.
Can I ask you to check this part?

 

 

( Jun Loader DSM 6.2.3 DS918+)

root@NAS4:~# dmesg | grep mpt

[    0.000000] Preemptible hierarchical RCU implementation.

[    2.757193] mptctl: Registered with Fusion MPT base driver

[    2.757195] mptctl: /dev/mptctl @ (major,minor=10,220)

[    2.779415] mpt3sas version 09.102.00.00 loaded

[    2.779544] mpt3sas 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control

[    2.781591] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8006160 kB)

[    2.836673] mpt2sas_cm0: MSI-X vectors supported: 1, no of cores: 4, max_msix_vectors: -1

[    2.836721] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 27

[    2.836723] mpt2sas_cm0: iomem(0x00000000f7940000), mapped(0xffffc90000260000), size(65536)

[    2.836724] mpt2sas_cm0: ioport(0x000000000000e000), size(256)

[    2.930127] mpt2sas_cm0: Allocated physical memory: size(7445 kB)

[    2.930129] mpt2sas_cm0: Current Controller Queue Depth(3307),Max Controller Queue Depth(3432)

[    2.930130] mpt2sas_cm0: Scatter Gather Elements per IO(128)

[    2.975727] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03), BiosVersion(07.39.02.00)

[    2.975729] mpt2sas_cm0: Protocol=(

[    2.977740] mpt2sas_cm0: sending port enable !!

[    3.126435] Got empty serial number. Generate serial number from product.

[    4.502776] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x), phys(8)

[   10.380271] mpt2sas_cm0: port enable: SUCCESS

[   76.371203] Got empty serial number. Generate serial number from product.

[   76.381080] Got empty serial number. Generate serial number from product.

 

==================================================================================

 

CPU : i3-4160T
M/B : DT-H81DL GAIDA
LAN : RealTek r8111F DUAL
RAID : Dell Perc H310 , IT Mode
LOADER : Tinycore 0.4.5.1 Native Lastest Update
EXT : r8169, mpt3sas (yesterday Lastest Update)
DSM : DS3622xs,+ DSM 7.0.1

HDD : 4ea Sata Drives

 

 

root@box:/home/tc# ./rploader.sh ext broadwellnk-7.0.1-42218 add https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

 

jq: error (at <stdin>:1): Cannot iterate over null (null)

jq: error (at <stdin>:1): Cannot iterate over null (null)

Loader source : https://github.com/dogodefi/redpill-load.git Loader Branch : develop

Redpill module source : https://github.com/dogodefi/redpill-lkm.git : Redpill module branch : develop

Extensions :

Extensions URL :

TOOLKIT_URL : https://sourceforge.net/projects/dsgpl/files/toolkit/DSM7.0/ds.broadwellnk-7.0.dev.txz/download

TOOLKIT_SHA : 0d9edca67d9e7e14c2529bbb58341b623936124d5264f71f1e4acbacf3ea202d

SYNOKERNEL_URL : https://sourceforge.net/projects/dsgpl/files/Synology NAS GPL Source/25426branch/broadwellnk-source/linux-4.4.x.txz/download

SYNOKERNEL_SHA : d3e85eb80f16a83244fcae6016ab6783cd8ac55e3af2b4240455261396e1e1be

COMPILE_METHOD : toolkit_dev

TARGET_PLATFORM       : broadwellnk

TARGET_VERSION    : 7.0.1

TARGET_REVISION : 42218

REDPILL_LKM_MAKE_TARGET : dev-v7

KERNEL_MAJOR : 4

MODULE_ALIAS_FILE= modules.alias.4.json

Checking Internet Access -> OK

Cloning into 'redpill-lkm'...

remote: Enumerating objects: 1142, done.

remote: Counting objects: 100% (1142/1142), done.

remote: Compressing objects: 100% (588/588), done.

remote: Total 1142 (delta 792), reused 869 (delta 547), pack-reused 0

Receiving objects: 100% (1142/1142), 464.83 KiB | 2.19 MiB/s, done.

Resolving deltas: 100% (792/792), done.

Cloning into 'redpill-load'...

remote: Enumerating objects: 867, done.

remote: Counting objects: 100% (334/334), done.

remote: Compressing objects: 100% (256/256), done.

remote: Total 867 (delta 168), reused 98 (delta 41), pack-reused 533

Receiving objects: 100% (867/867), 24.36 MiB | 8.80 MiB/s, done.

Resolving deltas: 100% (374/374), done.

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

[OK]

[#] ========================================== pocopico.mpt2sas ==========================================

[#] Extension name: mpt2sas

[#] Description: Adds Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt2sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

root@box:/home/tc# ./rploader.sh build broadwellnk-7.0.1-42218

jq: error (at <stdin>:1): Cannot iterate over null (null)

jq: error (at <stdin>:1): Cannot iterate over null (null)

Loader source : https://github.com/dogodefi/redpill-load.git Loader Branch : develop

Redpill module source : https://github.com/dogodefi/redpill-lkm.git : Redpill module branch : develop

Extensions :

Extensions URL :

TOOLKIT_URL : https://sourceforge.net/projects/dsgpl/files/toolkit/DSM7.0/ds.broadwellnk-7.0.dev.txz/download

TOOLKIT_SHA : 0d9edca67d9e7e14c2529bbb58341b623936124d5264f71f1e4acbacf3ea202d

SYNOKERNEL_URL : https://sourceforge.net/projects/dsgpl/files/Synology NAS GPL Source/25426branch/broadwellnk-source/linux-4.4.x.txz/download

SYNOKERNEL_SHA : d3e85eb80f16a83244fcae6016ab6783cd8ac55e3af2b4240455261396e1e1be

COMPILE_METHOD : toolkit_dev

TARGET_PLATFORM       : broadwellnk

TARGET_VERSION    : 7.0.1

TARGET_REVISION : 42218

REDPILL_LKM_MAKE_TARGET : dev-v7

KERNEL_MAJOR : 4

MODULE_ALIAS_FILE= modules.alias.4.json

Checking Internet Access -> OK

Checking if a newer version exists on the repo -> Version is current

Redpill sources already downloaded, pulling latest

Already up to date.

Loader sources already downloaded, pulling latest

Already up to date.

Using static compiled redpill extension

Looking for redpill for : ds3622xsp_42218

Getting file https://raw.githubusercontent.com/pocopico/rp-ext/master/redpill/releases/redpill-4.4.180plus-broadwellnk.tgz

Extracting module

Getting file https://raw.githubusercontent.com/pocopico/rp-ext/master/redpill/src/check-redpill.sh

Got redpill-linux-v4.4.180+.ko

Testing modules.alias.4.json -> File OK

------------------------------------------------------------------------------------------------

It looks that you will need the following modules :

 

 

Found VGA Controller : pciid 8086d0000041e  Required Extension : i915

Searching for matching extension for i915

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

######################################################################## 100.0%

[OK]

[#] ========================================== pocopico.mpt3sas ==========================================

[#] Extension name: mpt3sas

[#] Description: Adds LSI MPT Fusion SAS 3.0 Device Driver Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt3sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3622xsp_42218 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

Found SAS Controller : pciid 1000d00000072  Required Extension : mpt3sas

Searching for matching extension for mpt3sas

Found matching extension :

"https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json"

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

######################################################################## 100.0%

[OK]

[#] ========================================== pocopico.r8169 ==========================================

[#] Extension name: r8169

[#] Description: Adds RealTek RTL-8169 Gigabit Ethernet driver Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/r8169

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3622xsp_42218 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

Found Ethernet Interface : pciid 10ecd00008168 Required Extension : r8169

Searching for matching extension for r8169

Found matching extension :

"https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json"

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

######################################################################## 100.0%

 

[!] Extension is already added (index exists at /home/tc/redpill-load/custom/extensions/pocopico.r8169/pocopico.r8169.json). For more info use "ext-manager.sh info pocopico.r8169"

 

*** Process will exit ***

Found Ethernet Interface : pciid 10ecd00008168 Required Extension : r8169

Searching for matching extension for r8169

Found matching extension :

"https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json"

------------------------------------------------------------------------------------------------

Starting loader creation

Checking user_config.json : Done

Cache directory OK

[#] Checking runtime for required tools... [OK]

[#] Updating extensions...

 

[#] Checking runtime for required tools... [OK]

[#] Adding new extension from https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json...

[#] Downloading remote file https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

[OK]

[#] ========================================== dogodefi.mpt3sas ==========================================

[#] Extension name: mpt3sas

[#] Description: Adds LSI MPT Fusion SAS 3.0 Device Driver Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/dogodefi/mpt3sas

[#] Software author: https://github.com/dogodefi

[#] Update URL: https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json

[#] Platforms supported: fs6400_42218 ds3622xsp_42218 ds918p_42218 rs3617xsp_42218

[#] =======================================================================================

 

[#] Checking runtime for required tools... [OK]

[#] Updating dogodefi.mpt3sas extension...

[#] Downloading remote file https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension dogodefi.mpt3sas index is already up to date

[#] Updating pocopico.mpt2sas extension...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension pocopico.mpt2sas index is already up to date

[#] Updating pocopico.mpt3sas extension...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension pocopico.mpt3sas index is already up to date

[#] Updating pocopico.r8169 extension...

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/r8169/rpext-index.json to /home/tc/redpill-load/custom/extensions/_new_ext_index.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Extension pocopico.r8169 index is already up to date

[#] Updating pocopico.r8169 extension... [OK]

[#] Checking runtime for required tools... [OK]

[#] Updating ds3622xsp_42218 platforms extensions...

[#] Downloading remote file https://raw.githubusercontent.com/dogodefi/mpt3sas/offical/releases/ds3622xsp_42218.json to /home/tc/redpill-load/custom/extensions/_ext_new_rcp.tmp_json

############################################################################################################################################################################################################## 100.0%

 

[#] Filling-in newly downloaded recipe for extension dogodefi.mpt3sas platform ds3622xsp_42218

[#] Downloading remote file https://github.com/dogodefi/mpt3sas/raw/offical/releases/mpt3sas-4.4.180plus-broadwellnk.tgz to /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/mpt3sas-4.4.180plus-broadwellnk.tgz

############################################################################################################################################################################################################## 100.0%############################################################################################################################################################################################################## 100.0%

[#] Verifying /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/mpt3sas-4.4.180plus-broadwellnk.tgz file... [OK]

[#] Unpacking files from /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/mpt3sas-4.4.180plus-broadwellnk.tgz to /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/... [OK]

[#] Downloading remote file https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt3sas/src/check-mpt3sas.sh to /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/check-mpt3sas.sh

############################################################################################################################################################################################################## 100.0%

[#] Verifying /home/tc/redpill-load/custom/extensions/dogodefi.mpt3sas/ds3622xsp_42218/check-mpt3sas.sh file... [OK]

[#] Successfully processed recipe for extension dogodefi.mpt3sas platform ds3622xsp_42218

[-] The extension pocopico.mpt2sas was found. However, the extension index has no recipe for ds3622xsp_42218 platform. It may not be

[-] supported on that platform, or author didn't updated it for that platform yet. You can try running

[-] "ext-manager.sh update" to refresh indexes for all extensions manually. Below are the currently known information about

[-] the extension stored locally:

[#] ========================================== pocopico.mpt2sas ==========================================

[#] Extension name: mpt2sas

[#] Description: Adds Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt2sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

[-] Failed to update recipe for pocopico.mpt2sas extension for platform ds3622xsp_42218. The script will terminate as you do not

[-] have previously downloaded recipe which can be used if download fails. Try again later. If problem

[-] persists contact the extension packer for support (displayed below)

[#] ========================================== pocopico.mpt2sas ==========================================

[#] Extension name: mpt2sas

[#] Description: Adds Support

[#] To get help visit: <todo>

[#] Extension preparer/packer: https://github.com/pocopico/rp-ext/tree/main/mpt2sas

[#] Software author: https://github.com/pocopico

[#] Update URL: https://raw.githubusercontent.com/pocopico/rp-ext/master/mpt2sas/rpext-index.json

[#] Platforms supported: ds3617xs_25556u2 ds918p_41890 ds3617xs_42218 ds3615xs_25556u2 ds3617xs_25556 ds918p_25556 ds3615xs_41222 ds3617xs_42218u2 ds3615xs_25556 ds918p_42218 ds3615xs_42218

[#] =======================================================================================

 

[!] Cannot continue due to previous errors (see above)

 

*** Process will exit ***

[!] Failed to update all extensions for ds3622xsp_42218 platform - see errors above

 

*** Process will exit ***

FAILED : Loader creation failed check the output for any errors

root@box:/home/tc#

 

Platforms with kernel versions 4.x have mpt2sas combined in MPT3SAS. You dont have to add mpt2sas. Just use MPT3SAS.

 

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...